So You Want to Hash a Password…

Posted on
1

Congratulations. You’re thinking about protecting a password; a concept that well-known1 sites, to this day2, fail3 to comprehend.

Choose an established, vetted algorithm (SHA-256 would suffice), include a salt (we’ll explain this a bit later), hash the password. Get rid of the plaintext password. Done. See how simple that was? There’s even Open Source code4 to help you with more complex issues.

But once you’ve set foot on the path of hashing passwords you might be tempted to make the hash Even Better. An apparently common idea is that if you hash a password once, hashing it twice makes it more secure. Being “more secure” is a commendable goal, but beware the wild beasts of cryptography, for they are subtle and quick to…well, you should be able to finish that thought.5

Repeating encryption or hashing algorithm isn’t a bad idea, it’s just not fully thought-through idea. First, we need a paragraph or three to catch everyone up on hashing and brute force attacks:

A cryptographic hash function takes an arbitrary-length input and produces a fixed-length output that has no statistical relation to the input. Consequently, a password like friend becomes an unintelligible string like 97823jnsndf234.6 An important property of a cryptographic hash function is that it’s irreversible (information is lost, similar to a lossy compression algorithm). No algorithm exists to turn the output 97823jnsndf234 back into friend. Alternately, an encryption function turns friend into mellon and if you know the encryption scheme and a key, then you know how to turn mellon back into friend. AES is an example of an encryption function.

Cracking a hashed password requires effort on the part of the attacker. This effort, or work factor, represents the time to execute a single hash function multiplied by the expected number of guesses required to find the correct input to the hash function. For example, an attacker might try all six-letter strings such as friena, frienb, frienc until finally hashing the guess of friend and observing that the output matches the reference hash, 97823jnsndf234.

Trying all six-letter lowercase combinations of the English alphabet requires 308,915,776 guesses (26 characters to the 6th power). This is actually a relatively small number in the age of multi-core behemoths and GPU trickery. If a single hash function takes 1 microsecond to execute on a particular system, then the complete brute force will take about 5 minutes. If you pass each input through the hash function N times, then you increase the work factor by N. With N = 100 the six-character attack would take close to 9 hours. The attacker is going to get the password eventually, but now it will take N times longer.

Notice that the previous equation only cared about the time required to execute a single hash function. From this perspective it doesn’t matter if the hash algorithm produces a 128 bit or 512 bit output. It only matters how long it takes to obtain the output. (We’re only talking about hashing and repeated hashing here; bit lengths and algorithm selection still have important security implications for other reasons and against other attacks.)

Here is a simplified explanation of how a repeated hash function fails to universally improve the work factor to brute force a value. The input plaintexts are marked P (with Greek letter subscripts). This brief examples uses 10 iterations of a lossy hash function. The output of each intermediate hash is marked H with a numeric subscript. The final hash iteration is marked C with a subscript corresponding to the original plaintext.

The following line shows how the final value for an input plaintext is achieved:

Pα -> H1 -> H2 -> H3 -> H4 -> H5 -> H6 -> H7 -> H8 -> H9 -> Cα

A different input should produce a different final value:

Pβ -> H43 -> H44 -> H45 -> H46 -> H47 -> H48 -> H49 -> Cβ

A problem occurs when the original plaintext has a collision with one of the intermediate or final hash values. For example, what if Pγ and H7 have the same result when passed into the hash function? You have an overlapping sequence from the end of Pα’s chain:

Pγ -> H8 -> H9 -> H10 -> H11 -> H12 -> H13 -> H14 -> H15 -> H16 -> Cγ

A more pathological case happens when a sequence overlaps significantly:

Pδ -> H44 -> H45 -> H46 -> H47 -> H48 -> H49 -> H50 -> Cδ

The way an attacker would exploit these artifacts is by creating some chain reference tables, much like a rainbow table. Yet in this case, the chain reference is used to skip rounds. For example, given an input plaintext Px, if the first hash round is H13 and the table has a precomputed chain with an H13 in it, then the attacker can fast-forward 10 steps (or however many steps have been precomputed) to get the Cx.

This case for this Time-Memory-Trade-Off (TMTO) attack chains didn’t present any math or probability calculations to back up these assertions. If you’re about to dismisse this attack based on the lack of hard evidence (in this article), consider something else about repeated rounds: they do not introduce additional entropy. Consequently, each round might actually weaken the entropy of the initial input despite the increased work factor due to additional rounds. In a worst case scenario, this dilution of entropy might lead to collisions that make a brute force search even easier.

Repeated hashing does not increase entropy (the “difficulty” of the initial password), it only increases the work factor. Repeated hashing of iloveyou doesn’t make the password any harder to guess, just longer to get there.

Think of it in terms of the attacker’s dictionary. The attacker has a pre-defined list of common passwords, from iloveyou to KAR120C. Neither the hashing algorithm nor the number of repetitions has any impact on this dictionary. Those only affect the amount of time required for the attacker to cycle through the dictionary.

The common theme for using cryptosystems is to first look for implementations that conform to a standard7 rather than creating something you think is new, novel, and unique.

In the case of repeated encryption, you should turn to RFC 2898 for the Password-Based Key Derivation Function 2 (PBKDF2).8 PBKDF2 inserts an iteration counter to prevent “chain” attacks. In other words, the attacker must perform every encryption stage. At a minimum, the iteration prevents the attacker from shortcutting rounds using a TMTO trick.

Where repeated rounds increase the attacker’s work factor, salts defeat other precomputation (a.k.a Rainbow table) attacks. A salt is merely a number of bytes (like a string, though it need not be) prefixed or suffixed to a password.

Salting passwords affects the composition of the attacker’s dictionary. Rather than trying the password me+galadriel the attacker must include a salt, which makes it somethinglongbefore-me+galadriel. Salts don’t make the dictionary bigger, they make the dictionary specific to the salt. The idea here is that all of the effort put forth to crack a password with a particular salt cannot be reused to crack the same password with a different salt — the brute force must begin anew. The hash for somethinglongbefore-me+galadriel is completely different from anotherstringinfrontof-me+galadriel. This is the primary way to prevent another TMTO attack, usually referred to as a rainbow table.

If you want a recommendation on the length of a salt, 19 is a nice, mystical number.9

Every measure you take to encrypt and obfuscate the password reduces the risk should the web site’s password store be stolen. (There’s quite a bit of precedent for such things.)

However, everything you do to protect the password in the database (or wherever it is stored) has no bearing on a multitude of other attack vectors, including the database itself.

Imagine a SQL injection attack that sets every user’s password to the hash of a password known to the attacker. What would you rather do? Download the entire DB over a period of several minutes or change every account to a password you know? These approaches have different goals: obtaining original passwords are likely re-used across email, banking, and other sites whereas setting a known password gives immediate access to the site at the expense of blatant activity more likely to be noticed.

Imagine a scenario where the attacker is able to modify the login page so cleartext passwords are stored to a file or shuffled off to another web site.

The focus on encrypting the password and preserving its confidentiality is laudable. However, too much focus takes away from the more immediate threat of brute forcing the login form itself. The work factor to crack a short password like ncc1701 might be measured in days or weeks depending on the method of encryption. On the other hand, the attacker may have a list of the site’s users (or have a reliable way of generating likely user names). In this scenario, the attacker targets the login page with a static password (ncc1701) and cycles through the user list.

Once again, there’s precedent of success for this approach such as against our high-profile friend Twitter. In 2009 a hacker cracked the long (more than the mystical “8 character minimum”), but unsophisticated password happiness for an account that had permissions to reset passwords for any other account.10

Clearly, it didn’t matter how well happiness had been kept secret, encrypted, obfuscated, and otherwise concealed. There were no limits on how many times the login page could be requested for brute forcing the account. Furthermore, the password protection for every other account was moot since the hacker now had access to an admin account from which he could take over any other. The only apparent good news in this scenario is that, while several accounts were compromised, the original passwords to those accounts were not. This is possibly negligible consolation, but important none the less considering the prevalence of password re-use across web sites.

By all means, put some effort into hashing passwords using well-established techniques. You’ll be adding to the work factor of anyone trying to crack the passwords should the password store ever be extracted from the site.

On the other hand, you may be increasing your own work factor with over-engineered solutions for password protection at the expense of other protections — like preventing SQL injection or rate limiting authentication points.

Here’s an additional note I made in the comments, but should highlight in the article:

For comparison, WPA2 uses PBKDF2 with the SSID of the network as a salt, a 256-bit key, HMAC-SHA1 for the algorithm, and 4096 iterations.

If you trying to figure out “what’s best” for hashing a password, consider WPA2 as the reference metric. For example, your hashing should generate a work factor of N times the work factor for WPA2 where N is your degree of paranoia that WPA2 is easily broken.

If you chose a double-digit N “just because”, then why would you ever use a wireless network (phone or Wi-Fi, GSM A5/3 or WPA2, etc.)? It’s much more likely someone will be able to sniff your encrypted traffic than they’ll ever get your hashed passwords. In fact, GSM’s A5/X algorithms have reported attacks. Seems like another reason to layer encryption, such as always using HTTPS.

For another comparison, the OSX File Vault apparently uses PBKDF2 with 1000 iterations. (Although it’d be nice to have a more detailed reference.)

=====

1 http://risky.biz/sosasta
2 http://blogs.wsj.com/digits/2011/06/08/some-top-apps-put-data-at-risk/
3 http://www.pcworld.com/article/229316/lulz_boat_hacks_sonys_harbor_faq.html#tk.hp_fv
4 http://www.keyczar.org/
5 If you’re not well-read in crypto you should at least be well read in fiction. After all, the security of a home-grown cryptosystem is closer to fiction.
6 It’s not necessary to think of any specific hash function at this point, but if you want a more concrete example, “friend” is hashed to the hexadecimal string “8e8b4d64f704c7a6aa632a7e6c2024e4f9fed79caac319e6bb7754db587e6f58″ using the SHA-256 algorithm.
7 http://csrc.nist.gov/groups/ST/hash/index.html
8 http://www.ietf.org/rfc/rfc2898.txt
9 Read the Dark Tower series by Stephen King, books V and VII in particular. The series also has one of the greatest first lines in a book, “The man in black fled across the desert, and the gunslinger followed.”
10 http://www.wired.com/threatlevel/2009/01/professed-twitt/

Big in Japan

Posted on
Reply

It’s not quite a Spinal Tap moment, but here’s a curious translation via Google.

Here’s the text from the original article1:

“Given the types of hacks that made the news in the last 12 months it’s not surprising that SQL Injection is high on the list,” Mike Shema, engineering lead for the Qualys Web application scanning service told InternetNews.com. “What is surprising is that the countermeasures to SQL injection are well-known, effective, and available in all of the major programming languages used in web apps — for at least half a decade.”

And the output after putting a Japanese version2 of the article through Google translate:

Mr. Mike Shema He has served as an engineering lead in Qualys vulnerability management for Web applications, said in an interview as follows. “Given the type of hacking made headlines during the past 12 months, that’s up to the top of the list of SQL injection is not surprising.’s Surprising is to measure at least 5 years SQL injection is not well known, effective, and it is in a state that can be used in all major programming languages used in Web Applications”

I love the fact that my cynical observation of Advanced Persistent Ignorance was turned on its head to clearly explain three reasons why SQL injection survives to this day:

  • it’s not well known,
  • it’s effective,
  • and (my favorite part) it can be used in all major programming languages.

It sounds so much better that way!

=====

1 http://www.esecurityplanet.com/features/article.php/3936581/SQL-Injection-Most-Dangerous-Software-Error.htm
2 http://japan.internet.com/webtech/20110630/4.html

Will the Real APT Please Stand Up?

Posted on
Reply

The Advanced Persistent Threat (APT) is now competing with Cyberwar for reign as security word(s) of the year. It would have been nice if we had given other important words like HTTPS1 or Prepared Statements their chance to catch enough collective attention to drive security fixes. Alas, we still deal with these fundamental security problems due to Advanced Persistent Ignorance. (I noted2 previously that you can only defeat Advanced Persistent Ignorance with CAKE3.) It’s not wrong to seek out examples of APT, but it helps to have an idea about its nature. Otherwise, we risk seeing the APT boogeyman everywhere.

Threats have agency. They are persons (or even natural events like earthquakes and tsunamis) that take action against your assets (information, network, etc.). An XSS vulnerability in an email site isn’t a threat — the person trying to hijack your account with it is. With this in mind, the term APT helpfully self-describes two important properties:

  • the threat is persistent
  • the threat is advanced

Persistence is uncomplicated. The threat actor has a continuous focus on the target. This doesn’t mean around-the-clock port scanning just waiting for an interesting port to pop up. It means active collection of data about the target as well as development of tools, techniques, and plans once a compromise is attained. Persistent implies patience in searching for “simple” vulns and enumerating resources vulnerable to more sophisticated exploits.

A script-kiddie joyriding the Internet with sqlmap4 or metasploit5 looking for anything to attack may be persistent, but the persistence is geared towards finding a vuln rather than finding a vuln in a specific target. It’s the difference between a creepy guy stalking his ex versus a creepy guy hanging out in a bar waiting for someone to get really drunk.

The advanced aspect of a threat leads to more confusion than its persistent aspect. An advanced threat may still exploit simple vulns (e.g. SQL injection). The advanced nature need not even be the nature of the exploit (e.g. using a tool like sqlmap). What may be advanced is the leverage of the exploit. Remember, the threat agent most likely wants to do more than grab passwords from a database. With passwords in hand it’s possible to reach deeper into the target network, steal information, cause disruption, and establish more permanent access than waiting for another buffer overflow to appear.

Stolen passwords are one of the easiest backdoors and the most difficult to detect. Several months ago RSA systems were hacked. Enough information was allegedly stolen that observers at the time imagined it would enable attackers to spoof or otherwise attack SecurID tokens and authentication schemes.

Now it seems those expectations have been met with not one6, but two7 major defense contractors reporting breaches that apparently used SecurID as a vector.

At this point I’m out of solid technical examples of APT. But I don’t want you to leave without a good understanding of what an insidious threat looks like. Let’s turn to the metaphor and allegory of television influenced by the Cold War.

Specifically, The Twilight Zone season 2, episode 28, “Will the Real Martian Please Stand Up” written by the show’s creator, Rod Serling.

Spoilers ahead. I insist you watch the episode before reading further. It’ll be 25 minutes of entertainment you won’t regret.

The set-up of the show is that a Sheriff and his deputy find possible evidence of a crashed UFO, along with very human-like footprints leading from the forested crash site into town.

The two men follow the tracks to a diner where a bus is parked out front. They enter the diner and start to ask if anyone’s seen someone suspicious. You know, like an alien. The bus driver explains the bus is delayed by the weather and they had just stopped at the diner. The lawmen scan the room, “Know how many you had?”

“Six.”

In addition to the driver and the diner’s counterman, Haley, there are seven people in the diner. Two couples, a dandy in a fedora, an old man, and a woman. Ha! Someone’s a Martian in disguise!

What follows are questions, doubt, confusion, and a jukebox. With no clear evidence of who the Martian may be, the lawmen reluctantly give up and allow everyone to depart. The passengers reload the bus8. The sheriff and his deputy leave. The bus drives away.

But this is the Twilight Zone. It wouldn’t leave you with a such a simple parable of paranoia; there’s always a catch.

The man in the fedora and overcoat, Mr. Ross, returns to the diner. He laments that the bus didn’t make it across the bridge. (“Kerplunk. Right into the river.”)

Dismayed, he sits down at the counter, cradling a cup of coffee in his left hand. The next instant, with marvelous understatement, he pulls a pack of cigarettes from his overcoat and extracts a cigarette — using a third hand to retrieve some matches.

We Martians (he explains) are looking for a nice remote, pleasant spot to start colonizing Earth.

Oh, but we’re not finished. Haley nods at Mr. Ross’ story. You see, the inhabitants of Venus thought the same thing. In fact, they’ve already intercepted and blocked the Ross’ Martian friends in order to establish a colony of their own. Haley smiles, pushing back his diner hat to reveal a third eye in his forehead.

That, my friends, is an advanced persistent threat!

=====

1 http://mashable.com/2011/05/31/https-web-security/

2 http://www.deadliestwebattacks.com/2011/04/advanced-persistent-ignorance.html

3 Continuous Acquisition of Knowledge and Experience

4 http://sqlmap.sourceforge.net/

5 http://www.metasploit.com/

6 http://www.reuters.com/article/2011/05/27/us-usa-defense-hackers-idUSTRE74Q6VY20110527

7 http://www.wired.com/threatlevel/2011/05/l-3/

8 The counterman rings up their bills, charging one of the $1.40 for his 14 cups of coffee. I’m not sure which is more astonishing — drinking 14 cups or paying 10 cents for each one.

Or Was it Sindarin?

Posted on
1

I have a new article1 on Mashable regarding the importance of having https:// in front of the web sites you visit.

I finished that article and its linguistic metaphor a few days before coming across an article2 on El Reg that describes research3 showing the feasibility of identifying language patterns over encrypted channels.

One goal of an encryption algorithm is to create diffusion of the original content in order to camouflage the content’s structure. For example, diffusion applied to a long English text, say one of Iain M. Bank’s novels, would reduce the frequency of the letter ‘e’ from the most common letter to (ideally) an equally common frequency within the ciphertext. The confusion property of an encryption algorithm would do something like replace every letter ‘e’ with the letter ‘z’, but that wouldn’t affect how frequently the letter appears — hence the need for diffusion.

There have been similar analyses of SSL and SSH in the past that demonstrated it was possible to infer the length of the encrypted content (which might reveal the length of a password even if the content is not known) or guess whether content was HTML or an image. The Skype analysis is a fantastic example of looking for structure within an encrypted stream and making inferences from those observations.

=====
1 http://mashable.com/2011/05/31/https-web-security/
2 http://www.theregister.co.uk/2011/05/26/bypassing_skype_crypto/
3 http://www.cs.unc.edu/~amw/resources/hooktonfoniks.pdf

The Futility of Web Pen Testing

Posted on
2

I previously lamented the death of web scanners1 so it’s only fair that to turn this nihilistic gaze to the futility of manual web security testing. This isn’t to say it’s impossible to perform a comprehensive review of a web app. The problem lies in repeating that review after developers modify the app. Or the problem of obtaining consistent results from different testers. Or if we continue to look for problems: Repeating an in-depth review of one app across the few thousand that might exist in an organization.

It’s not controversial to state the web apps need to be tested. The trick is trying to keep up with the pace of development for a single web app, compounded by the pace at which new apps arise. Security testers must deal with the fear that a once-secure app might be crippled by the introduction of a new feature or a change that inadvertently breaks an old one. With this in mind, the fundamental challenges of manual pen testing is managing the effort required to maintain one site’s security and scaling that effort to thousands.

If we look at recent compromises, and accept reasoning from an anecdotal basis, we see simple attacks succeeding against huge, well-known web sites. It’s not like Google, Twitter, Facebook, and similar don’t understand security, don’t have budgets for security testing, or don’t have people testing their apps. Google actively encourages ethical testing against several of its properties.2 This nod to explicit permission to find vulns isn’t a concession to the impossibility of writing secure code; it’s a nod towards the difficulty of scale in manually testing large, complex web applications. It’s also interesting to see the vulns considered worthy of reward versus those deemed cruft or a “vector for petty mischief”.3 This highlights the minefield within the subjectivity of risk when terms like risk, attack, threat, and impact are ambiguously defined or too-broadly applied.

Consider the Sony Playstation Network (PSN) breach4 that compromised user data “by hacking into an application server behind a Web server and two firewalls”.5 This attack6 is a topical example of the impact of (what seems to be7) straight-forward web vulns like SQL injection. One could wonder whether the vulnerable web site ever received security testing. If so, the quality of the test must be called into question, especially if the alleged attack vector was so simple. On the other hand, we could ruminate on reasons why the site wasn’t tested. Missed and forgotten because the site was so old? Assumed it had been tested before? Too many other sites considered more important needed to be tested first? We could go on, but the point of the exercise is to express the difficulty of maintaining web security, not second-guessing situations we know too little about.

The presence of a vulnerability is usually indisputable. On the other hand, its risk or exploitation impact often falls to debate. SQL injection is a clear example of a vuln that should be addressed immediately rather than arguing if the vuln exposes an empty database or encrypted credit card numbers. SQL injection flaws are due to fundamentally improper programming. Even if an exploit is questionable, there’s bad code sitting on the server that should be fixed.

CSRF is a different matter, especially depending on how it manifests. Arguments over risk ratings too often devolve into opinions biased by imaginative threats rather than focus on the situation. (For example, mistakenly considering a CSRF countermeasure broken because it can be bypassed by XSS or incorrectly assuming CSRF tokens prevent sniffing attacks.)

The details of web vulns differ enough that it’s hard, and perhaps unwise, to assign each a static risk. Efforts like CVSS8 try to bring consistency and terminology to describing software vulns, but the scoring systems seems infrequent within web security. Fortunately, the risk calculations can be avoided without great loss if you treat vulns like the software defects they are. Assign priorities using methodologies already familiar to your dev team. And if the effort required to fix a bug is greater than the effort required to prove it’s really, really, truly a problem, then have a discussion about impact and risk.

Manual testing has a high degree of variance in quality and coverage. I don’t make these statements without being blameless. In the past I’ve written a chapter or two that omitted an attack variant or didn’t highlight something well enough. These are things I’ve tried to address in revised editions and on this web site. My point is that manual tests have an unavoidable bias in focus that depends on the person conducting them. Rather than passing judgement on quality, this is more of a judgement on coverage and that inevitable human factor of making mistakes. (After all, lots of vulns boil down to mistakes in code, albeit fairly consequential ones.)

CSRF arrived on the OWASP Top 10 in 2007. How many people were testing their web apps before that year? (To be fair, how many apps were being compromised that way? Especially when SQL injection remains(!) so much easier.) I like to pick on CSRF because the vuln is easily misunderstood and its purported impacts and countermeasures vary immensely.

There are two methodologies for addressing test coverage: blackbox (review the deployed app) and whitebox (review the app’s code). Blackbox testing rarely requires knowledge of the app’s underlying language. Although cases like PHP show that prior knowledge can be helpful because configuration settings influence code execution. The same code may run securely on one server and be wide open under a different configuration. Blackbox testing is the easiest path because it requires nothing more than a browser to begin. The snag is that blackbox testing won’t necessarily find the dusty corners of the web site where an insecure link or form is hiding.

Then why not just step towards a code review where every corner can be checked? A pen tester could spend an hour investigating different XSS vectors against a web page whereas reviewing the page’s source code could provide higher confidence whether it’s secure. After all, not every security fix is securely fixed.9

The drawback is that whitebox testing reduces the population of testers capable of finding security problems. This exacerbates the problem of scale in matching testers to web apps. The tester needs to have good comprehension of the app’s programming language in addition to security concepts. Someone very good at reviewing Java may miss a vuln in C#. Knowledge of good software design translates well between languages. Yet the problem remains that too few people have too many sites to review.

As with the article on web scanner mortality, this one requires a caveat. There will be a vocal group hurling cliched invectives against the idea that manual testing is useless, never requires tools, and is a worthless endeavor. None of those were asserted here. In fact, manual testing must be part of any exhaustive security testing. Humans have the ability to analyze the design of a web site, where an automated scanner largely focuses on implementation problems. Humans also possess the creative thinking required to turn QA’s use cases into abuse and misuse cases that bypass security.

Web QA testing groups are likely already overloaded with the combinatorial craziness inherent to reviewing UIs. Yet this is a perfect step for identifying security problems alongside bugs in features. Tools like Selenium10 would be prime platforms for security testing. Yet how many times did a web pen test produce findings, provide a PDF, then leave? Why haven’t Selenium scripts (or anything similar) become a lingua franca of pen test results?

One of the biggest changes necessary to manual testing is translating the hands-on tests that a human performs into a script that someone else (or a tool) can repeat.11 Treating pen tests as snapshots in time misses the opportunity to build a repository of security knowledge. Rather than just manage vulns, a group could manage the techniques and scripts used to find such vulns. This not only enables a one-time security review to become repeatable, but the quality of testing can improve.

The pessimism of this article and the previous one isn’t intended to be a capitulation to web vulnerabilities. By identifying the fundamental challenges to security testing it’s possible to start thinking of creative ways to solve them. It’s important to understand a problem well to avoid branching off into solutions that address false issues or have too narrow of a focus. In future articles we’ll turn the tables on this bleak landscape and look at the effective ways to apply automation and manual testing to web sites.

=====

1 http://www.deadliestwebattacks.com/2011/05/death-of-web-scanners.html

2 http://googleonlinesecurity.blogspot.com/2010/11/rewarding-web-application-security.html

3 http://www.google.com/corporate/rewardprogram.html

4 http://us.playstation.com/support/answer/index.htm?a_id=2356

5 http://news.cnet.com/8301-31021_3-20058950-260.html?tag=mncol;txt

6 http://blog.us.playstation.com/2011/04/26/update-on-playstation-network-and-qriocity/

7 I’ve yet to find definitive explanations of the attack, so reserve a little skepticism for reports like this: http://news.cnet.com/8301-27080_3-20063789-245.html

8 http://nvd.nist.gov/cvss.cfm

9 http://blog.mindedsecurity.com/2010/09/twitter-domxss-wrong-fix-and-something.html

10 http://seleniumhq.org/ 11 Dinis Cruz recognized this type of problem and started the O2 project, https://www.owasp.org/index.php/OWASP_O2_Platform. However, O2 focuses more on the tools to implement repeatability rather than defining a grammar to describe vuln tests. Selenium is a similar tool that uses JavaScript to define tests that can be driven by one of several different programming languages; however, it does not have an explicit security bent.

JavaScript ViewState Parser

Posted on
2

I completed the first version of a JavaScript-based ViewState decoder.

The parser should work with most non-encrypted ViewStates. It doesn’t handle the serialization format used by .NET version 1 because that version is sorely outdated and therefore too unlikely to be encountered in any real situation.

I’m working on a version to decode encrypted ViewState. That version will require knowledge of the decryption key. (While creating a brute-forcer in JavaScript to guess the decryption key might be interesting from a development perspective, it’s utility is questionable and success improbable.)

The next step will be the ability to edit the ViewState contents and re-serialize it.

If you encounter any problems, feel free to ask questions or post troublesome ViewStates in the comments below.

A Spirited Peek into ViewState, Part II

Posted on
2

Our previous article1 started with an overview of the ViewState object. It showed some basic reverse engineering techniques to start deconstructing the contents embedded within the object. This article broaches the technical aspects of implementing a parser to automatically pull the ViewState apart.

We’ll start with a JavaScript example. The code implements a procedural design rather than an object-oriented one. Regardless of your design preference, JavaScript enables either method.

The ViewState must be decoded from Base64 into an array of bytes. We’ll take advantage of browsers’ native atob and btoa functions2 rather than re-implement the Base64 routines in JavaScript. Second, we’ll use the proposed ArrayBuffer3 data type in favor of JavaScript’s String or Array objects to store the unencoded ViewState. Using ArrayBuffer isn’t necessary, but it provides a more correct data type for dealing with 8-bit values.

Here’s a function to turn the Base64 ViewState into an array of bytes in preparation of parsing: 

function analyzeViewState(input) {  var inputLength = input.length;  var rawViewState = atob(input);  var rawViewStateLength = rawViewState.length;  var vsBytes = new Uint8Array(ArrayBuffer(rawViewStateLength));

  for(i = 0; i < rawViewStateLength; ++i) {    vsBytes[i] = rawViewState.charCodeAt(i);  }

  if(vsBytes[0] == 0xff && vsBytes[1] == 0x01) {    // okay to continue, we recognize this version    // starting parsing...    var i = 2;    while(i < vsBytes.length) {      i = parse(vsBytes, i);    }  }  else {    document.writeln("unknown format");  }}

The parse function will basically be a large switch statement. It takes a ViewState buffer, the current position in the buffer to analyze (think of this as a cursor), and returns the next position. The skeleton looks like this: 

function parse(bytes, pos) {  switch(bytes[pos]) {    case 0x64:  // EMPTY      ++pos;      break;    default:    // unknown byte      ++pos;      break;  }

  return pos;}

If you recall from the previous article, strings were the first complex object we ran into. But parsing a string also required knowing how to parse numbers. This is the function we’ll use to parse numeric values. The functional approach coded us into a bit of a corner because the return value needs to be an array that contains the decoded number as an unsigned integer and the next position to parse (we need to know the position in order to move the cursor along the buffer): 

function parseUInteger(bytes, pos) {  var n = parseInt(bytes[pos]) & 0x7f;

  if(parseInt(bytes[pos]) > 0x7f) {    ++pos;    var m = (parseInt(bytes[pos]) & 0x7f) << 7;    n += m;

    if(parseInt(bytes[pos]) > 0x7f) {      ++pos;      var m = (parseInt(bytes[pos]) & 0x7f) << 14;      n += m;    }  }

  ++pos;

  return [n, pos];}

With the numeric parser created we can update the switch statement in the parse function: 

var r = [0, 0];  switch(bytes[pos]) {    case 0x02:      ++pos;      r = parseUInteger(bytes, pos);      pos = r[1];      document.writeln("number: " + r[0]);      break;    ...

Next up is parsing strings. We know the format is 0x05, followed by the length, followed by the “length” number of bytes. Now add this to the switch statement: 

switch(bytes[pos]) {    ...    case 0x05:      ++pos;      r = parseUInteger(bytes, pos);      var size = r[0];      pos = r[1];      var s = parseString(bytes, pos, size);      pos += size;      document.writeln("string (" + size + "): " + s.replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;'));      break;    ...

The parseString function will handle the extraction of characters. Since we know the length of the string beforehand it’s unnecessary for parseString to return the cursor’s next position: 

function parseString(bytes, pos, size) {  var s = new String("");

  for(var i = pos; i < pos + size; ++i) {    s += String.fromCharCode(parseInt(bytes[i], 10));  }

  return s;}

We’ll cover two more types of objects before moving on to an alternate parser. A common data type is the Pair. As you’ve likely guessed, this is an object that contains two objects. It could also be called a tuple that has two members. The Pair is easy to create. It also introduces recursion.4 Update the switch statement with this: 

switch(bytes[pos]) {    ...    case 0x0f:      ++pos;      document.writeln("pair");      pos = parse(bytes, pos);  // member 1      pos = parse(bytes, pos);  // member 2      break;    ...

More containers quickly fall into place. Here’s another that, like strings, declares its size, but unlike strings may contain any kind of object: 

switch(bytes[pos]) {    ...    case 0x16:      ++pos;      r = parseUInteger(bytes, pos);      var size = r[0];      pos = r[1];      document.writeln("array of objects (" + size + ")");      for(var i = 0; i < size; ++i) {        pos = parse(bytes, pos);      }      break;    ...

From here you should have an idea of how to expand the switch statement to cover more and more objects. You can use this page5 as a reference. JavaScript’s capabilities exceed the simple functional approach of these previous examples; it can handle far more robust methods and error handling. Instead of embellishing that code, let’s turn our text editor towards a different language: C++.

Diving into C++ requires us to start thinking about object-oriented solutions to the parser, or at least concepts like STL containers and iterators. You could very easily turn the previous JavaScript example into C++ code, but you’d really just be using a C++ compiler against plain C code rather than taking advantage of the language.

In fact, we’re going to take a giant leap into the Boost.Spirit6 library. The Spirit library provides a way to create powerful parsers using clear syntax. (Relatively clear despite one’s first impressions.) In Spirit parlance, our parser will be a grammar composed of rules. A rule will have attributes related to the data type is produces. Optionally, a rule may have an action that executes arbitrary code.

Enough delay. Let’s animate the skeleton of our new grammar. The magic of template meta-programming makes the following struct valid and versatile. Its why’s and wherefore’s may be inscrutable at the moment; however, the gist of the parser should be clear and, if you’ll forgive some exaltation, quite elegant in terms of C++: 

template <typename Iterator>struct Grammar : boost::spirit::qi::grammar<Iterator>{  Grammar()    : Grammar::base_type(start)  {    using boost::spirit::qi::byte_;

    empty =     byte_(0x64);

    object =    empty             |  pair;

    pair =      byte_(0x0f)            >>  object            >>  object;

    version =   byte_(0xff)            >>  byte_(0x01);

    start =     version   // must start with recognized version            >>  +object;  // contains one or more objects  }

  qi::rule<Iterator>  empty,                      object,                      pair,                      start,                      version;};

We haven’t put all the pieces together for a complete program. We’ll put some more flesh on the grammar before unleashing a compiler on it. One of the cool things about Spirit is that you can compose grammars from other grammars. Here’s how we’ll interpret strings. There’s another rule with yet another grammar we need to write, but the details are skipped. All it does it parse a number (see the JavaScript above) and expose the value as the attribute of the UInteger32 rule.7 The following example introduces two new concepts, local variables and actions: 

template <typename Iterator>struct String : boost::spirit::qi::grammar<Iterator, qi::locals<unsigned> >{  String()    : String::base_type(start)  {    using boost::spirit::qi::byte_;    using boost::spirit::qi::omit;    using namespace boost::spirit::qi::labels;

    start =          omit[                (byte_(0x05) | byte_(0x1e))            >>  length                      [ _a = _1 ]          ]            >>  repeat(_a)[byte_]    ;  }

  UInteger32<Iterator>                        length;  qi::rule<Iterator, qi::locals<unsigned> >   start;};

The action associated with the length rule is in square brackets. (Not to be confused with the square brackets that are part of the repeat syntax.) Remember that length exposes a numeric attribute, specifically an unsigned integer. The attribute of a rule can be captured with the _1 placeholder. The local variable for this grammar is captured with _a. Local variables can be passed into, manipulated, and accessed by other rules in the grammar. In the previous example, the value of length is set to _a via simple assignment in the action. Next, the repeat parser takes the value of _a to build the “string” stored in the ViewState. The omit parser keeps the extraneous bytes out of the string.

Now we can put the String parser into the original grammar by adding two lines of code (highlighted in bold). That this step is so trivial speaks volumes about the extensibility of Spirit: 

...    object =    empty             |  my_string             |  pair;

    ...    start =     version   // must start with recognized version            >>  +object;  // contains one or more objects  }

  String<Iterator>    my_string;  qi::rule<Iterator>  empty,  ...

The String grammar introduced the repeat parser. We’ll use that parser again in the grammar for interpreting ViewState containers. At this point the growth of the grammar accelerates quickly because we have good building blocks in place: 

...    using boost::spirit::qi::byte_;    using boost::spirit::qi::repeat;

    container =            byte_(0x16)        >>  length          [ _a = _1 ]        >>  repeat(_a)[object]    ;

    empty =     byte_(0x64);

    object =    empty             |  pair;    ...  }

  String<Iterator>      my_string;  UInteger32<Iterator>  length;  qi::rule<Iterator, qi::locals<unsigned> >   container;  qi::rule<Iterator>    empty,  ...

This has been a whirlwind introduction to Spirit. If you got lost along the way, don’t worry. Try going through the examples in Spirit’s documentation. Then, re-read this article to see if the concepts make more sense. I’ll also make the sample code available to help get you started.

There will be a few surprises as you experiment with building Spirit grammars. For one, you’ll notice that compilation takes an unexpectedly long time for just a dozen or so lines of code. This is due to Spirit’s template-heavy techniques. While the duration contrasts with “normal” compile times for small programs, I find it a welcome trade-off considering the flexibility Spirit provides.

Another surprise will be error messages. Misplace a semi-colon or confuse an attribute and you’ll be greeted with lines and lines of error messages. Usually, the last message provides a hint of the problem. Experience is the best teacher here. I could go on about hints for reading error messages, but that would be an article on its own.

Between compile times and error messages, debugging rules might seem a daunting task. However, the creators of Spirit have your interests in mind. They’ve created two very useful aids to debugging: naming rules and the debug parser. The following example shows how these are applied to the String grammar. Once again, the change is easy to implement:

...    start =          omit[                (byte_(0x05) | byte_(0x1e))            >>  length                      [ _a = _1 ]          ]            >>  repeat(_a)[byte_]    ;  }

  start.name(“String”);  // Human readable name for the rule

  debug(start);  // Produce XML output of the parser’s activity

  UInteger32<Iterator>                        length;  qi::rule<Iterator, qi::locals<unsigned> >   start;};

As a final resource, Boost.Spirit has its own web site8 with more examples, suggestions, and news on the development of this fantastic library. You’ll also find that the Spirit mailing list9 is an active, helpful venue. And you’ll rarely have a Spirit-related question go unanswered on StackOverflow10.

It seems unfair to provide all of these code snippets without a complete code listing for reference or download. Plus, I suspect formatting restrictions may make it more difficult to read. Watch for updates to this article that provide both full code samples and more readable layout. Hopefully, there was enough information to get you started on creating your own parsers for ViewState or other objects.

In the next article in this series we’ll shift from parsing ViewState to attacking it and using it to carry our attacks past input validation filters into the belly of the web app. In the mean time, I’ll answer questions in the comments below. If you’d like to learn more about Spirit, let me know — I’d be happy to throw together more articles on the topic.

=====

1 http://www.deadliestwebattacks.com/2011/05/spirited-peek-into-viewstate-part-i.html

2 http://aryeh.name/spec/base64.html

3 Sadly, this isn’t yet exposed in Safari although WebKit (it’s “brain”) supports it. https://developer.mozilla.org/en/JavaScript_typed_arrays/ArrayBuffer

4 Finally an example of recursion that doesn’t mention the Fibonacci sequence!

5 http://www.deadliestwebattacks.com/p/viewstate-parsing.html

6 http://www.boost.org/doc/libs/release/libs/spirit/index.html

7 Spirit has pre-built parsers for many data types, including different types of integers. We need to use custom ones to deal with ViewState’s numeric serialization.

8 http://boost-spirit.com/. Also look at the presentation at http://objectmodelingdesigns.com/boostcon10/spirit_presentation.pdf

9 http://boost-spirit.com/home/info/mailing-list/

10 http://stackoverflow.com/