Category: web scanner evaluation

I come here not to bury web application scanners, but to praise them.¹ And then bury them a bit. Perhaps just up to the neck. On the beach. At low tide.

Web application has historically been challenging, even in the early days of so-called simple web sites with low complexity and little JavaScript. Such simple sites would have static HTML, gobs of links, and icons proclaiming “Best viewed in so-and-so screen size” or “Optimized for so-and-so browser”. Even though links were relatively easy to extract, scanners would run into deeply nested links, infinitely recursive links, or redirect loops.

Setting aside web security scanning, the necessary feat of QA testing web sites has been variably difficult, non-existent, or so manually intensive that it lags the pace of development. The challenges of automation aren’t specific to security, but security testing does impose some unique requirements and diverges in distinct ways from QA testing.

Reading through a complete list of automation challenges would exceed the patience of many more than the 140-character crowd. Consider these happy few:²

• Efficiently crawl sites with thousands or tens of thousands (or more!) links
• Populate form fields correctly in order to obtain new links and exercise workflows
• Interact with the site as a browser does, in other words the ability to deal with JavaScript and DOM manipulation
• Identify multi-step workflows and non-idempotent requests

I hesitate to arrange these in significance or difficulty, or to explain which have reasonable solutions. Instead, I want to focus on the implications that web application design and technology trends have for automation.

Web sites are progressing towards more dynamic, interactive UIs near indistinguishable from the neolithic, disconnected age of desktop apps and strongly divorced from the click-reload pages of the 90’s. That web sites have nifty UIs isn’t news if the first ones that come to mind fall into the Alexa Top 20 or so. Those are the easy examples of early pioneers of this trend. Once you look lower on the Alexa list or drop off it entirely to consider large organizations’ internal networks you’ll find sites still designed for IE6 or that have never heard of the XHR object.

The trend towards dynamic UIs won’t affect legacy sites, which is good news for scanners that choose to remain in the past. Sites rely heavily on JavaScript (and CSS and the DOM) to attain a modern look. Such browser-heavy apps will only increase as HTML5 creeps closer to being officially standardized.³ Consequently, scanners must be able to handle complex JavaScript libraries rather than employ regex-based fakery.

Using pattern matching to statically parse the content from these dynamic sites may work to a degree, but such a technique misses certain classes of vulnerabilities. DOM-based XSS is one example. While in theory it’s possible to create regex patterns to search for particular manifestations of this problem, regexes are too limited to be of real use or produce so much noise that they threaten to increase the workload of manually testing a site. HTML5’s Cross-Origin Request Sharing (CORS) is another example where analyzing JavaScript and the XHR object require a browser. Count the Web Storage API⁴ as yet another example.

Regardless of your perspective on the pace of web security’s evolution, web application technologies have been changing quickly. It’s unlikely that the SQL database will disappear from the traditional web site stack, but the NoSQL⁵ movement will require a new set of vuln tests largely unrelated to traditional SQL injection. There are no publicly known examples of “NoSQL injection” attacks, nor even clear ideas on what an attack of that kind would look like. Yet that’s no reason to avoid applying security theory to the practice of testing NoSQL-backed web sites.

Single sign-on (SSO) solutions should eventually become more widely adopted. They alleviate the burden of managing and security passwords, which is evidently difficult to do right. (Compromised credentials from database attacks number in the millions.) The distrust⁶ of early solutions like Microsoft Passport⁷ (and its MS Wallet) and the Liberty Alliance⁸ has been forgotten in light of Facebook Connect, Google Account, OpenID, Twitter, and Yahoo! ID. (There’s possibly an SSO service for each letter of the alphabet even though they mostly use OAuth.) Privacy issues haven’t been forgotten, they’ve just been swept aside in the face of millions of users with accounts on one or more of these sites.

By this point, you might have forgotten that we were originally discussing automated web scanning. The implications of Single Sign-On is that scanners must be able to support them. Once again this boils down to robust browser emulation — or a lot of customization to different sites’ use of the SSO APIs.

Browser plugins have always been a hindrance to scanners. Not only do ActiveX-based plugins and Flash have their own inherent security risks, but the content generated from them is either rarely parseable or poorly parsed by tools other than the plugin itself. Many web developers and users would rejoice if HTML5 heralded the demise of plugins. Unfortunately, efforts like Google’s Native Client⁹ promise to bring back the era of write once, run oncewhere (in Chrome) rather than Java’s write once, run anywhere. To hijack the title of an excellent book (and graphic novel), it would be nice to relegate plugins to write nonce, run neverwhere. Until sites stick universally to HTML and JavaScript, scanners will need to handle plugin-based content that drives a site’s navigation.

Forward-looking site developers aren’t satisfied with HTTP. Now that they’ve been getting a taste of HTML5’s features, they’re turning their sights to the deficiencies in HTTP and HTTPS. This means scanners should start thinking about things like SPDY¹⁰, designed for network performance, and HSTS¹¹, designed for improved transport security. Few sites have adopted these, but considering those few sites include behemoths like Google and Paypal expect others to follow.

The acronym assault hasn’t yet finished. REST is the new SOAP (at least I think so, I’m not sure if SOAP ever caught on). I’ve noted elsewhere the security benefits of a well-defined separation between the server-side API and client-side HTML. As a reminder, a server-side API call that performs a single action (e.g. get list of foo) can be easier to examine and secure as opposed to a function that gets a list, rewrites the page’s HTML, and has to update other unrelated content.

In one way, the move towards well-defined APIs makes a scanner’s job easier. If it’s possible to fully enumerate a site’s functions and their associated parameters, then the scanner doesn’t necessarily have to crawl thousands of different links trying to figure out where the important, unique points of functionality are — it can pull this information from a documented API.

Alas, a raw list of API calls emphasizes a problem scanners already have: state context. You and I can review a list of functions, then come up with a series of security tests. For example, calling events.create with an XSS payload followed by a call to events.get to see if the payload was filtered, or calling admin.banUsers from a non-admin account to see if authorization controls are correctly enforced. A dull scanner, on the other hand, might make calls in a poorly chosen order. In a somewhat contrived example, the scanner might call events.get followed by auth.expireSession (which logs out the user). This causes any subsequent API call to fail (at least it should) if the call requires an authenticated user.

Before we finish, permit a brief aside to address the inevitable concern trolls. There’s a Don Quixote contingent fighting straw man arguments that automation is useless, unusable, disusable, will never replace manual testing, and so on. This article doesn’t aim to engage these distractions.¹² I can control quote-mining no more than I can raise the sun. This paragraph serves as a warning about taking statements out of context or twisting its intent. To be clear: I think a degree of automation is important, accurate, and scaleable. And possible. The goal is to accompany technology trends, not trail them.

HTML and HTTP may not have changed very much in the past decade, but the way web sites cobble them together surely has. As web apps grow into more complex UIs, scanners must more accurately emulate (or outright embed) a browser to make sure they’re not missing swaths of functionality. As APIs become more common, scanners must dive into stateful modeling of a site. And as new web specifications and protocols become widely adopted, scanners must avoid the laziness of dealing solely with HTTP and HTML4.

It’s twilight for the era of simple scripting and unsophisticated scanners. The coming tide of HTML5, new plugins, protocols, and complexity make for a long night. With luck, some scanners will survive until dawn.

=====

1 An uniambic inversion of Mark Anthony’s speech. http://shakespeare.mit.edu/julius_caesar/julius_caesar.3.2.html

2 An allusion to King Branagh’s speech to his bowmen, incidentally preventing St. Crispian’s descent into obscurity. http://shakespeare.mit.edu/henryv/henryv.4.3.html

3 Regardless of its draft status, all modern browsers support at least a few features of HTML5.

4 http://dev.w3.org/html5/webstorage/

5 http://nosql-database.org/

6 s/Microsoft/Google/g and fast-forward a decade. Plus ca change, eh? http://www.wired.com/politics/security/news/2001/08/46095

7 http://support.microsoft.com/kb/277759

8 http://projectliberty.org/

9 http://code.google.com/p/nativeclient/

10 http://dev.chromium.org/spdy

11 http://tools.ietf.org/html/draft-ietf-websec-strict-transport-sec-01

12 Another of Shakespeare’s Romans, Menenius, provides apt words, “…more of your conversation would infect my brain…” http://shakespeare.mit.edu/coriolanus/coriolanus.2.1.html

When web site owners want to measure how their visitors get from point A (say, the home page) to point B (such as finalizing a purchase), they might use a metric called click depth or link depth. This represents the number of clicks required to get from link A to link B. Sites strive to minimize this value so users may more efficiently perform actions without being distracted or frustrated — and consequently depart for other venues. The depth of a link also implies that popular or important pages should have lower values (i.e. “closer” to the home page, easier to find) than less important pages. This train of thought might make sense superficially, but this reasoning derails quickly for web scanners.

There’s merit to designing a web application, or any human interface, to have a high degree of usability. Minimizing the steps necessary to complete an action helps achieve this. Plus, your users will appreciate good design. Web application scanners are not your users, they don’t visit your web site and follow workflows that humans do.

Click depth for web scanning is useless. Pointless. It’s a long string of synonyms for pointless when used as a configuration option, doubly so when scanning web sites that use a JavaScript-driven UI or implement simple Search Engine Optimization (SEO) techniques.

There’s a long list of excuses why someone might want to rely on click depth as an option for web scanning: Links on the home page are more likely to be attacked, vulnerabilities with low click depth are easier to find, opportunistic attackers are the biggest threat, scans run faster. Basically, these arguments directly correlate link popularity with risk. The simple rejoinder is that all links have a depth of 1 in the face of automation. An attacker who invests effort into scripts that search for vulnerable links doesn’t care how deep a link is, just that the scripts finds one.

Whether the correlation of link popularity and risk rings true or not, having the scanner calculate the click depth is fundamentally incorrect. Visitors’ behavior influences a link’s popularity, not the calculation of a scanner. A superior approach would be to use analytics data to identify popular links, then feed that list to the scanner.

Another reason for click depth’s inutility is the positive trend in web application design to create richer, more interactive interfaces in the browser that use lighter-weight data requests back to the web site. This is reflected in the explosion of Ext JS, Prototype, YUI, and other JavaScript libraries designed to provide powerful UI features along with concise request/response handling using JSON and asynchronous requests. This also has the effect of flattening web applications in terms of the number of clicks required to accomplish tasks. Even more significantly it has the effect of separating links into two conceptual buckets: one for links that show up in the browser bar and another for “behind the scenes” links used for API requests. Both link buckets are important to security testing, but the idea of click depth among them has little meaning.

SEO techniques can also flatten a page’s apparent link depth. A technique common to e-commerce sites is to create a long list of links on the home page that reach deep into the site’s product catalog. It’s not uncommon to see several dozen links at the bottom of a home page that point to a different product pages ad nauseum. (The purpose of which is to make sure search engines find all of the site’s products so users looking for a particular shade of Unicorn-skin rugs will find that site over all others.) This sets an artificially low depth for many, many pages. A human is unlikely to care about the slew of links, but a scanner won’t know the difference.

We’ve reached three reasons so far: Automated scanning gives every link an effective click depth of 1, browser-heavy sites have flat APIs, and SEO techniques further reduce apparent link depth. In spite of this, click depth appeared at some point in scanner designs, an OWASP project makes it a point of evaluation (among several poor criteria), and users often ask for it.

One understandable motivation behind click depth is trying to obtain some degree of depth or breadth in the coverage of a web site’s functionality. Notice that coverage of a site’s functionality differs from coverage of the site’s links. Sites might contain vast numbers of links that all exercise the same, small number of code paths. It’s these code paths in the web application where vulnerabilities appear. This sense of click depth actually intends to convey coverage. It’s highly desirable to have a scanner that avoids making dozens of redundant requests, following recursive links, or getting stuck in redirect loops. A good scanner handles such situations automatically rather than burdening the user with a slew of configuration options that may not even have a bearing on the problem.

Designing a web application scanner is easy. A good design requires a few sentences; a great design might need two paragraphs or so. It’s easy to find messages on e-mail lists that describe the One True Way to scan a web site.

Implementing a scanner is hard. The core of a web vulnerability scanner performs two functions: find a link, test that link. The task of finding links falls to a crawling engine. The crawler must be fundamentally strong, otherwise links will be missed and a missed link is an untested link. Untested links lead to holes in the site coverage which raise uncertainty in the state of the site’s security. It’s rarely necessary to hit every link of a web site in order to adequately scan it. Security testing requires comprehensive coverage of the site’s functionality, which is different from covering every single link. A SQL injection vulnerability in the thread ID of a forum can be found by crawling a few sample discussion threads. It’s not necessary to fully enumerate 100,000 threads about nerfing warlocks or debating Mal vs. Kirk.

In addition to crawling strategies, scanners must also be able to crawl a site as an authenticated user. Maintaining an authenticated state requires coordinating several pieces of information (tracking the session cookie, avoiding logout links). But first the scanner must find and submit the login form.

Simple login forms have a text field, password field, and submit button. The HTML standard provides the markup to create these forms. The standard only defines syntax, not usage. This gives web developers leeway to abuse HTML through ignorance, inefficiency, and what can only be termed outright malice.

Consider the login form created by Sun’s OpenSSO Enterprise 8.0. The HTML roughly breaks down to the following:

<head>
<script language="JavaScript">
var defaultBtn = 'Submit';
var elmCount = 0;
/** submit form with default command button */
function defaultSubmit() {
  LoginSubmit(defaultBtn);
}
</script>
</head>
<body>
<script language="javascript">
elmCount++;
</script>
<form name="frm1" action="blank" onSubmit="defaultSubmit(); return false;" method="post">
User Name: <input type="text" name="IDToken1" id="IDToken1" value="" class="TxtFld">
</form>
<form name="frm2" action="blank" onSubmit="defaultSubmit(); return false;" method="post">
Password: <input type="password" name="IDToken2" id="IDToken2" value="" class="TxtFld">
</form>
<form name="Login" action="/login/UI/Login?AuthCookie=..."  method="post">
<script language="javascript">
if (elmCount != null) {
  for (var i = 0; i < elmCount; i++) {
    document.write("<input name=\"IDToken" + i + "\" type=\"hidden\">");
  }
  document.write("<input name=\"IDButton" + "\" type=\"hidden\">");
}
</script>
<input type="hidden" name="goto" value=" aHR0cHM6Ly93d3cuZGVhZGxpZXN0d2ViYXR0YWNrcy5jb20vc2VjcmV0L2xpbmsvaW4vYmFzZS82 NC8=">
<input type="hidden" name="SunQueryParamsString" value="">
<input type="hidden" name="encoded" value="true">
<input type="hidden" name="gx_charset" value="UTF-8">
</form>
</body>

So we have a single page with three forms, two of which have no purpose other than to display a form field, and a final one with JavaScript whose sole purpose is to copy values from the other two forms into its own hidden fields. There’s also a single <script> tag dedicated to nothing more than incrementing an element counter. Not only might this offend the sensibilities of JavaScript developers who appreciate more programmatic approaches as found in JQuery or Prototype.JS, it also causes headaches for web security scanners. The two forms’ actions are “blank” and the onSubmit events always return false. That should at least inform the scanner that they shouldn’t be submitted directly – but that’s an assumption that might prove false since the site may go into an error state if it receives an incomplete set for form fields.
Even uglier login form patterns exist in the wild. In some cases the login form is wrapped within its own HTML element:

<html>
<head></head>
<body>
...other content
<div>
<html><form>
...
</form></html>
</div>
...other content
</body>
</html>

Some forms use unnamed input fields and programmatically enumerate them via JavaScript upon submission:

Username: <input type="text" value="">
Password: <input type="password" value="">

Then there’s the doPostBack function in .NET sites along with their penchant for multiple submit buttons (e.g. one for authentication, another for a search). Now the scanner has to identify the salient fields for authentication and hit the correct submit button; it’s no good to fill out the username and password only to submit the search button.

Sure, a user could manually coax the scanner through any of these login processes, but that places an unnecessary burden on the user’s time. This is less of a problem when dealing with a single web site, but becomes overwhelming when trying to scan a dozen or even hundreds of web applications.

These types of logins also highlight the difficulty scanners have with understanding the logic of a web page, let alone the logic of a group of pages or some workflow within the site.
It’s still possible to automate the login process for these forms, doing so requires customization at the expense of having a generic authentication mechanism. In the end, dealing with login pages often provides insight into the madness of HTML editing (it’s hard to call some of these methods programming) and the bizarre steps developers will take just to “make it work.”

Scanners should automate the crawl and test phases as much as possible. After all, it’s dangerous to tie too much of a scan’s effectiveness to the user’s knowledge of web security. It may not be every day that a web developer answers your question about the robots.txt file with, “I don’t know what that is,” but it’s a good idea to have a scanner that will be comprehensive and accurate regardless of whether the user knows the UTF-7 encoding for an angle bracket or wonders why web sites don’t just strip the alert function to prevent XSS attacks.

Web scanner evaluations collect metrics by comparing scan results against a (typically far too small) field of test sites. One quick way to build the test field might be to collect intentionally vulnerable sites from the Web. That approach, though fast, does a disservice to the scanners and more importantly the real web applications that need to be scanned. After all, does your web application really look like WebGoat or the latest HacmeFoo? Does it even resemble an Open Source application like WordPress or phpBB?

The choice of targets highly influences results — not necessarily in terms of a scanner’s perceived accuracy or capabilities, but whether those properties will translate to a completely different¹ site. This doesn’t imply that a scanner that fails miserably against a test site will miraculously work against your own. There should always be a baseline that establishes confidence, however meager, in a scanner’s capabilities. However, success against one of those sites doesn’t ensure equal performance against a site with completely different workflows and design patterns.

Peruse Alexa’s top 500 web sites for a moment. They differ not only in category — adult and arts, science and shopping — but in technology and design patterns. Category influences the types of workflows that might be present. In terms of interaction, a shopping site looks and works differently from a news site. A search portal works differently than an auction site. I, of course, don’t know how the adult sites work, other than they make lots of money and are often laced with malware (even tame ones). Dealing with workflows in general creates problems for web scanners.

Remember that scanners have no real understanding of a site’s purpose nor visibility into its source code. Scan reports provide identification of, not insight into, a vulnerability. For example, a cross-site scripting bug might be due to a developer who neglected the site’s coding guidelines and used an insecure print() function rather than a centralized, well-tested print_safe() version. Perhaps the site has no centralized library for displaying user-supplied data in a secure manner. The first case was a mistake in a single page, the latter points to a fundamental problem that won’t disappear after fixing one page. Identifying underlying security problems remains the purview of manual testing and analysis — along with great hourly rates.

Design patterns also influence navigation and workflows, whether pages are built statically, incrementally with calls to document.write(), or dynamically with event-based calls to xmlHttpRequest objects that populate the DOM. These patterns influence, for example, how forms are put together, a common one being .NET forms with VIEWSTATE fields and doPostBack() functions. Some sites are driven by a single index page that handles dozens of actions based on query parameters. Other patterns put more emphasis on JavaScript execution in the browser, whether in the style of frameworks like JQuery, Prototype or YUI, or more tightly integrated client/server frameworks like DWR. Other technologies might affect the site’s response to invalid input or non-existent pages.

A web application scanner should be an agnostic to the mishmash of acronyms and technologies that underpin a site. As long as the server communicates via HTTP and throws up (in varying senses) HTML, then the scanner should be able to look for vulnerabilities. Neither HTTP nor HTML change in any appreciable way whether a site uses PHP-On-Dot-Rails or other language of the jour. If pages render in a web browser, then it’s a web site. Such is the ideal world of scanners and unicorns.

Cracks in this world emerge when a scanner has to deal with pages written by programmers whose experience ranges from burning toast to creating JavaScript singleton prototypes. Commercial scanners have been available for a decade, in turn predated by Open Source tools like whisker. Since the universal scanner doesn’t exist, it’s necessary to create an evaluation or look at metrics that more closely match the web applications to be secured.

How well does the scanner scale? If it can deal with a site that has 1,000 links, what happens when it hits 10,000? It might melt LAN ports scanning a site one network hop away that serves pages with an average size of 1KB only to blow up against a site with a network latency of a few hundred milliseconds and page sizes on the average of a few hundred kilobytes.

Scaleability also speaks to the time and resources necessary to scan large numbers of sites. This point won’t concern users who are only dealing with a single property, but some organizations have to deal with dozens, hundreds, or possibly a few thousand web properties. Manual approaches, although important for in-depth assessments, do not scale. For its part automation needs to enhance scaleability, not create new hindrances. This applies to managing scan results over time as well managing the dozens (or more!) scans necessary. If a scanner requires a high-end desktop system to scan a single site in a few hours, then simple math tells you how long it will take to test N sites that each require M hours on average to complete. You could parallelize the scans, but buying and maintaining more systems induces additional costs for hardware, software, and maintenance.

A test site (preferably plural) must provide areas where scanners can be evaluated for accuracy. A good field of test sites includes applications that exercise different aspects of web security scanning, especially those most relevant to the types of sites to be secured. Web sites require different levels of emphasis on

• Reliance on client-side execution (e.g. lots of JavaScript)
• Customized error handling
• Large numbers of links, including highly redundant areas like product pages
• Large page sizes
• Varying degrees of server responsiveness (there’s a big difference between scanning a site supported by load-balanced web farm and bludgeoning a Mac Mini with hundreds of requests per second)

A good web security scanner adapts to the peculiarities of its targets, from mistyped markup that browsers silently fix to complex pages rife with bizarre coding styles. HTML may be an established web standard, but few sites follow standard ways of creating and using it. Not only does this challenge web scanner developers, it complicates the process of in-depth scanner comparisons. No example site with a few dozen links will ever be representative or fully expose the pros and cons of a scanner.

—–
¹ At this point we’ve passed the Twitterati’s mystical 140-character limit four times over. So, for those who choose TLDR as a way of life, stop reading and go enjoy something completely different
instead.

This is the first in a series of essays describing suggested metrics for evaluating web application security scanners.

Accuracy measures the scanner’s ability to detect vulnerabilities. The basic function of a web scanner is to use automation to identify the same, or most of the same, vulnerabilities as a web security auditor. Rather than focus on the intricate details of specific vulnerabilities, this essay describes two major areas of evaluation for a scanner: its precision and faults.

An opinion piece covering all you ever wanted to know. Maybe not everything, but it provides a basis for the different challenges of automating web attacks. It might even sound familiar to a previous post…

Read a Q&A on some of the fundamental challenges of automating a web application scan. (I wrote the A, not the Q.)