Codex Securum, Obiter Dictum

In the past, you have come here for truth. I now give you law.

Science fiction author Arthur C. Clarke succinctly described the wondrous nature of technology in what has come to be known as Clarke’s Third Law (from a letter published in Science in January 1968):

Any sufficiently advanced technology is indistinguishable from magic.

The sentiment of that law can be found in an earlier short story by Leigh Brackett, “The Sorcerer of Rhiannon,” published in Astounding Science-Fiction Magazine in February 1942:

Witchcraft to the ignorant . . . Simple science to the learned.

With those formulations as our departure point, we can now turn towards crypto, browser technologies, and privacy.

The Latinate Lex Cryptobellum:

Any sufficiently advanced cryptographic escrow system is indistinguishable from ROT13.

Or in Leigh Brackett’s formulation:

Cryptographic escrow to the ignorant . . . Simple plaintext to the learned.

A few Laws of Browser Plugins (somewhat like the fonts of dis-knowledge):

Any sufficiently patched Flash is indistinguishable from a critical update.

Any sufficiently patched Java is indistinguishable from Flash.

A few Laws of Browsers:

Any insufficiently patched browser is indistinguishable from malware.

Any sufficiently patched browser remains distinguishable from a privacy-enhancing one.

For what are browsers but thralls to Laws of Ads:

Any sufficiently targeted ad is indistinguishable from chance.

Any sufficiently distinguishable person’s browser has tracking cookies.

Any insufficiently distinguishable person has privacy.

Mike’s law of writing on schedule:

Any sufficiently delivered manuscript is indistinguishable from overdue.

Which leads us to the foundational Zeroth Law of Deadliest Web Attacks:

Any sufficiently popular post is indistinguishable from truth.

Please share!



p.s. I highly recommend Gary Westfahl’s Science Fiction Quotations: From the Inner Mind to the Outer Limits should you wish to discover more authors and their books to explore.

The Rank Decay Contingency

The idea: Penalize a site’s ranking in search engine results if the site suffers a security breach.

Now, for some background and details…

In December 2013 Target revealed that it had suffered a significant breach that exposed over 40 million credit card numbers. A month later it upped the count to 70 million and noted the stolen information included customers’ names, mailing addresses, phone numbers, and email addresses.

Does anybody care?

Or rather, what do they care about? Sure, the media likes stories about hacking, especially ones that affect millions of their readers. The security community marks it as another failure to point to. Banks will have to reissue cards and their fraud departments be more vigilant. Target will bear some costs. But will customers really avoid it to any degree?

Years ago, in 2007, a different company disclosed its discovery of a significant breach that affected at least 40 million credit cards. Check out the following graph of the stock price of the company (TJX Holdings) from 2006 to the end of 2013.

TJX Price 2006-2014
TJX Price 2006-2014

Notice the dip in 2009 and the nice angle of recovery. The company’s stock didn’t take a hit until 2009 when TJX announced terms of its settlement. The price nose-dived, only to steadily recover as consumers stopped caring and spent money (amongst any number of arbitrary reasons, markets not being as rational or objective as one might wish).

Consider who bears the cost of breaches like these. Ultimately, merchants pay higher fees to accept credit cards, consumers pay higher fees to have cards. And, yes, TJX paid in lost valuation over a rather long period (roughly a year), but only when the settlement was announced — not when the breach occurred. The settlement suggests that lax security has consequences, but a breach in and of itself might not.

Truth of Consequences

But what if a company weighs the costs of a breach as more favorable than the costs of increasing security efforts? What if a company doesn’t even deal with financial information and therefore has no exposure to losses related to fraud? What about companies that deal in personal information or data, like Snapchat?

Now check out another chart. The following data from Quantcast shows daily visitors to a lyrics site. The number is steady until one day — boom! — visits drop by over 60% when the site is relegated to the backwaters of search results.

RapGenius Quantcast Measure

Google caught the site (Rap Genius) undertaking sociopathic search optimization techniques like spreading link spam. Not only does spammy, vapid content annoy users, but Google ostensibly suffers by losing users who flee poor quality results for alternate engines. (How much impact it has on advertising revenue is a different matter.) Google loses revenue if advertisers care about where the users are or they perceive the value of users to be low.

The two previous charts have different time scales and measure different dimensions. But there’s an underlying sense that they reflect values that companies care about.

Rank Decay

Think back to the Target breach. (Or TJX, or any one of many breaches reported over the years, whether they affected passwords or credit cards.)

What if a penalty affected a site’s ranking in search results? For example, it could be a threshold for the “best” page in which it could appear, e.g. no greater than the fourth page (where pages are defined as blocks of N results, say 10). Or an absolute rank, e.g. no higher than the 40th entry in a list.

The penalty would decay over time at a rate, linear or exponential, based on any number of mathematical details. For example, a page-based penalty might decay by one page per month. A list-based penalty might decay by one on a weekly basis.

The decay rate could be influenced by steps the site takes to remediate the underlying problem that led to the breach, improvements to a privacy policy, fines, or covering costs related to fraud as a result of the breach.

If the search engines drives a significant portion of traffic — that results in revenue or influences valuation — then this creates an incentive for the site to maintain strong security. It’s like PCI with different teeth. It might incentivize the site to react promptly to breaches. At least one hopes.

But such a proposal could have insidious consequences.

Rank Implications

Suppose a site were able to merely buy advertising to artificially offset the rank penalty? After a breach you could have a search engine that’d love to penalize the “natural” ranking of a site only to rake in money as the site buys advertising to overcome the penalty. It’s not a smart idea to pay an executioner per head, let alone combine the role with judge and jury.

A penalty that a company fears might be one for which it suppresses the penalty’s triggers. Keeping a breach secret is a disservice to consumers. And companies subject to the S.E.C. may be required to disclose such events. But rules (and penalties) need to be clear in order to minimize legal maneuvering through loopholes.

The proposal also implies that a search engine has a near monopoly on directing traffic. Yes, I’m talking about Google. The hand waving about “search engines” is supposed to include sites like Yahoo! and Bing, even DuckDuckGo. But if you’re worried about one measure, it’s likely the Google PageRank. This is a lot of power for a company that may wish to direct traffic to its own services (like email, shopping, travel, news, etc.) in preference to competing ones.

It could also be that the Emperor wears no clothes. Google search and advertisements may not be the ultimate arbiter of traffic that turns into purchases. Strong, well-established sites may find that the traffic that drives engagement and money comes just as well from alternate sources like social media. Then again, losing any traffic source may be something no site wants to suffer.

Target is just the most recent example of breaches that will not end. Even so, Target demonstrated several positive actions before and after the breach:

– Transparency — periodic updates on breach details, remediation steps, complaint process.

– A clear privacy policy — written in accessible language (i.e. avoids a legal style that, however accurate, may be too dense, misleading, or ambiguous), including a summary of changes.

Thankfully, there were no denials, diminishing comments, or signs of incompetence on the part of Target. Breaches are inevitable for complex, distributed systems. Beyond prevention, goals should be minimizing their time to discovery and maximizing their containment.

And whether this rank idea decays from indifference or infeasibility, its sentiment should persist.

Advanced Persistent Ignorance

The biggest threat to modern web applications is developers who exhibit Advanced Persistent Ignorance. Developers rely on all sorts of APIs to build complex software. This one makes code insecure by default. API is the willful disregard of simple, established security designs.

First, we must step back into history to establish a departure point for ignorance. This is just one of many. Almost seven years ago on July 13, 2004 PHP 5.0.0 was officially released. Importantly, it included this note:

A new MySQL extension named MySQLi for developers using MySQL 4.1 and
later. This new extension includes an object-oriented interface in addition
to a traditional interface; as well as support for many of MySQL’s new
features, such as prepared statements.

Of course, any new feature can be expected to have bugs and implementation issues. Even with an assumption that serious bugs would take a year to be worked out, that means PHP has had a secure database query mechanism for the past six years.1

The first OWASP Top 10 list from 2004 mentioned prepared statements as a countermeasure.2 Along with PHP and MySQL, .NET and Java supported these, as did Perl (before its popularity was subsumed by buzzword-building Python and Ruby On Rails). In fact, PHP and MySQL trailed other languages and databases in their support for prepared statements.

SQL injection itself predates the first OWASP Top 10 list by several years. One of the first summations of the general class of injection attacks was the 1999 Phrack article, Perl CGI problems.3 SQL injection was simply a specialization of these problems to database queries.

So, we’ve established the age of injection attacks at over a dozen years old and reliable countermeasures at least six years old. These are geologic timescales for the Internet.4

There’s no excuse for SQL injection vulnerabilities to exist in 2011.

It’s not a forgivable coding mistake anymore. Coding mistakes most often imply implementation errors — bugs due to typos, forgetfulness, or syntax. Modern SQL injection vulns are a sign of bad design. For six years, prepared statements have offered a means of establishing a fundamentally secure design for database queries. It takes actual effort to make them insecure. SQL injection attacks could still happen against a prepared statement, but only due to egregiously poor code that shouldn’t pass a basic review. (Yes, yes, stored procedures can be broken, too. String concatenation happens all over the place. Never the less, writing an insecure stored procedure or prepared statement should be more difficult than writing an insecure raw SQL statement.)

Maybe one of the two billion PHP hobby projects on Sourceforge could be expected to still have these vulns, but not real web sites. And, please, never in sites for security firms. Let’s review the previous few months:

November 2010, military web site.

December 2010, open source code repository web site.

February 2011, HBGary Federal. Sauron’s inept little brother. You might have heard about this one.

February 2011, Dating web site.

March 2011, Umm…speechless. Let’s move on.

April 2011, web security firm Barracuda (and this)

Looking back on the list, you might first notice that The Register is the of SQL injection vulns. (That is, in addition to a fine repository of typos and pun-based innuendos. I guess they’re just journalists after all, hackers don’t bother with such subtleties.)

The list will expand throughout 2011.

XSS is a little more forgivable, though no less embarrassing. HTML injection flaws continue to plague sites because of implementation bugs. There’s no equivalent of the prepared statement for building HTML or HTML snippets. This is why the vuln remains so pervasive: No one has figured out the secure, reliable, and fast way to build HTML with user-supplied data. This doesn’t imply that attempting to do so is a hopeless cause. On the contrary, JavaScript libraries can reduce these problems significantly.

For all the articles, lists, and books published on SQL injection one must assume that developers are being persistently ignorant of security concepts to such a degree that five years from now we may hear yet again of a database hack that disclosed unencrypted passwords.

If you’re going to use performance as an excuse for avoiding prepared statements then you either haven’t bothered to measure the impact, you haven’t understood how to scale web architectures, and you might as well turn off HTTPS for the login page so you can get more users logging in per second. If you have other excuses for avoiding database security, ask yourself if it takes longer to write a ranting rebuttal or a wrapper for secure database queries.

There may in fact be hope for the future. The rush to scaleability and the pious invocation of “Cloud” has created a new beast of NoSQL data stores. These NoSQL databases typically just have key-value stores with grammars that aren’t so easily corrupted by a stray apostrophe or semi-colon in the way that traditional SQL can be corrupted. Who knows, maybe security conferences will finally do away with presentations on yet another SQL injection exploit and find someone with a novel, new NoSQL Injection vulnerability.

Advanced Persistent Ignorance isn’t limited to SQL injection vulnerabilities. It has just spectacularly manifested itself in them. There are many unsolved problems in information security, but there are also many mostly-solved problems. Big unsolved problems in web security are password resets (overwhelmingly relying on e-mail) and using static credit card numbers to purchase items.

SQL injection countermeasures are an example of a mostly-solved problem. Using prepared statements isn’t 100% secure, but it makes a significant improvement. User authentication and password storage is another area of web security rife with errors. Adopting a solution like OpenID can reduce the burden of security around authentication. As with all things crypto-related, using well-maintained libraries and system calls are far superior to writing your own hash function or encryption scheme.

Excuses that prioritize security last in a web site design miss the point that not all security has to be hard. Nor does it have to impede usability or speed of development. Crypto and JavaScript libraries provide high-quality code primitives to build sites. Simple education about current development practices goes just as far. Sometimes the state of the art is actually several years old — because it’s been proven to work.

The antidote to API is the continuous acquisition of knowledge and experience. Yes, you can have your cake and eat it, too.


1 MySQL introduced support for prepared statements in version 4.1, which was first released April 3, 2003.
4 Perhaps a dangerous metaphor since here in the U.S. we still have school boards and prominent politicians for whom a complete geologic time spans a meager 4,000 years. Maybe some developers enjoy using ludicrous design patterns.

Some tips on answering certain interview questions

I often ask a job candidate how familiar they are with a particular security or web topic in order to gauge the technical starting point of a question. For example, if the interviewee is somewhat familiar (or perhaps feels likes a 3 on a scale of 1 to 5) with HTTP I’ll dig into whether GET and POST are any different from a security perspective and expect to hear some distinctions between session (i.e. non-persistent) cookies and session IDs as they relate to authentication. Someone less familiar might receive a question on workflows (e.g. what types of threats might there be to a check-out process) that focuses on critical thinking rather than knowledge of the protocol. Someone more familiar should be able to name the pros and cons of possible CSRF defenses. By doing this I hope to set the candidate at ease in order to start a conversation that can grow in a direction where I can try to (quickly) figure out their level of knowledge.

In any case, basing your security experience on the SQL injection article from Wikipedia won’t get you very far.

Tip zero: “I don’t know” would have worked fine. Really, it would. I’d rather have an interesting conversation about something else that you do know.

Tip one: When reading verbatim from a web page try to avoid changing the inflection of your voice from a conversational tone to a lecturing one. While I may not hear you frantically typing keywords into a search engine, I can hear the switch from extemporaneous ideas to textual regurgitation.

Tip two: If you’re going to double-down on first-hand knowledge of SQL injection vulns by claiming to know different ways to exploit them, don’t repeat the initial mistake by reading the “Blind SQL Injection” description from the same article.

Tip three: Don’t quote Wikipedia. Although I find the article on SQL injection poorly stated and I never use it as a reference, it’s a safe bet that I’m familiar enough to recognize its wikispeak. At least try something from the article’s References or External Links sections. You might even get bonus points for quoting something from Chapter Three of The Book — everyone has a weak spot.

Now, I too often err on the side of politeness so here’s the deal: If I ask a suspiciously-leading follow-up question stated as, “Imagine you have a table called users and you want to see if the username Ralph exists. What do you do?” then just pretend that you’re driving into a tunnel and hang up. Whatever you do don’t answer, “I would use a SELECT one divided by zero”. Otherwise, you’re going into the same bin as the persons who listed knowing Pi to the Feynman point as a key skill1 and the ability to learn any programming language in four hours2.

1 Trust me, I love the joke, but with a bold geek step like that you’ve got to have some other skills to back it up.
2 My quibble is the ability to do something useful besides writing “Hello, world.”

30% of the OWSAP Top 10 not common, only 1 not hard to detect

One funny point about the new OWASP Top 10 Application Security Risks is that 30% of them aren’t even common. The “Weakness Prevalence” for Insecure Cryptographic Storage (A7), Failure to Restrict URL Access (A8), and Unvalidated Redirects and Forwards (A10) is rated uncommon. Other than the apparent disconnect of putting effort into rare risks, these three points highlight the challenge of producing an unbiased list. (Yes, risk is complex and difficult to quantify. Ignore the document’s What’s My Risk? section at your peril.) The list is influenced by the experience of people who perform penetration tests, code reviews, or conduct research on web security. It’s not put together by the actual attackers who compromise web sites for fun and profit; they’re busy stealing your stuff.

Insecure Cryptographic Storage is rated with an uncommon prevalence and difficult to detect. One of the reasons it’s hard to detect is that back-end storage schemes can’t be reviewed by blackbox scanners (i.e. web scanners) nor can source code scanners point out these problems other than by indicating misuse of a language’s crypto functions. So, one interpretation is that insecure crypto is uncommon only because more people haven’t discovered or revealed such problems. Yet not salting a password hash is one of the most egregious mistakes a web app developer can make and one of the easier problems to fix. (The practice of salting password hashes has been around since epoch time was single digits.)

It’s also interesting that insecure crypto is the only one on the list that’s rated difficult to detect. XSS (A2) is “very widespread” and “easy” to detect, but maybe it’s only very widespread because it is so trivial to find. People might simply be lazy. Or maybe it is very widespread because it’s actually not easy to find — at least assuming web developers are looking for it in the first place. If more developers, or more tools, found XSS proactively then it wouldn’t be so widespread. (Although I still insist that XSS vulns are the cockroaches of the web.)

Want one item that probably carries the most risk as evidenced by real-world attacks? Password guessing. The Apache Foundation recently suffered an attack that relied on brute forcing a password. (Apache’s network had a similar password-driven intrusion in 2001. They’re mentioned because of the clarity of the event write-ups, not that they’re inherently insecure.) In 2009 Twitter provided happiness to another password guesser. There are plenty more examples. Knowing the password to an account is the best way to pilfer a user’s data and gain unauthorized access to a site. The only markers for an attacker using valid credentials are behavioral patterns – time of day the account was accessed, duration of activity, geographic source of the connection, etc. The attacker doesn’t have to use any malicious characters or techniques that could trigger XSS or SQL injection countermeasures. Broken Authentication and Session Management (A3) cover many more items than brute force attacks.

Incidentally, the impact of a compromised password is similar to the affect CSRF (A5) can have on a site. CRSF changes the nature of the attack so that the victim’s browser, which still has a valid session cookie regardless of whether the site is currently open in a window, makes an authenticated request on the attacker’s behalf. The target web site neither sees the attacker’s traffic nor even gets a whiff of the attacker’s activity because all of the interaction occurs between the victim’s browser and the target site.

CSRF serves as a good example of the changing nature of the web security industry. CSRF vulnerabilities have existed as long as the web. The attack takes advantage of the fundamental nature of HTML and HTTP whereby browsers automatically make requests to servers defined in img, iframe, or a handful of other tags. The servers in different tags need not be related and, since the attack doesn’t need to parse the response from the server, the browser’s Same Origin Rule doesn’t protect the user. CSRF hopped on the Top 10 list’s revision in 2007, three years after the list’s first appearance. It’s doubtful that CSRF vulnerabilities were any more or less prevalent over that three year period (or even the before 2000). Its inclusion was a nod to having a better understanding and appreciation of the vulnerability.

On the other hand, it’s not clear how useful a CSRF exploit was (or is) to attackers. If the attack is predicated on leading a victim to a site with a booby-trapped iframe or img, then why not just pepper the browser with malware rather than count on the confluence of a logged in victim hitting the page? Set aside the infamous web-app-worm-that-will-not-be-named and Jermiah Grossman’s cool Gmail hack. How many high impact web attacks have used CSRF?

Actually, that’s a bit of an unfair question. CSRF is really useful against sites that millions of people visit (Amazon, Ebay, Facebook, MySpace, Twitter, Yahoo!, this web site) because the number of users and the length of time they spend logged in their accounts raises the chance of finding a victim enormously. For other sites, CSRF might just not be useful enough to a hacker that finds something else like SQL injection. There’s a big difference between finding a victim from a pool of a few hundred million and a few hundred thousand.

The question also points out the confirmation bias of security researchers. Now that CSRF is in vogue people start to look for it everywhere. Security conferences get more presentations about advanced ways to exploit the vulnerability, even though real-world attackers seem fine with the returns on guessing passwords and seeding web pages with malware. Take a look at HTML injection (what everyone else calls XSS). Injecting script tags into a web page via an unchecked parameter dates back to the beginning of the web. Yet the early exploits didn’t do nearly as many cool things like tunneling HTTP traffic through XSS.

Before you shrug off this discussion of CSRF as hand waving with comments like, “But I could hack site Foo by doing Bar and then make lots of money,” consider what you’re arguing: A knowledgeable or dedicated attacker will find a useful exploit. Risk can be based on many things, including Threat Agents (to use the Top 10 list’s term). Risk increases under a targeted attack – someone actively looking to compromise a specific site or the users’ information the site contains. If you want to add an “Exploitability” metric to your risk calculation, keep in mind that ease of exploitability is often related to the threat agent and tends to be a step function: it might be hard to figure out the exploit in the first place, but anyone can run a 42-line Perl script that automates the attack.

This is why the Top 10 list should be a starting point to defining security practices for your web site, but it shouldn’t be the end of the road. Even the OWASP site admonishes readers to use the list for awareness rather than policy. So, if you’ve been worried about information leakage and improper error handling since 2007 don’t think the problem has gone away because it’s not on the list in 2010. And if you don’t think anyone cares about the logic problems within your site…well, just hope they haven’t been reading about it somewhere else.

(UFO label = Update For OWASP)

Observations on Larry Suto’s Paper about Web Application Security Scanners

Note: I’m the lead developer for the Web Application Scanning service at Qualys and I worked at NTO for about three years from July 2003 — both tools were included in this February 2010 report by Larry Suto. Never the less, I most humbly assure you that I am the world’s foremost authority on my opinion, however biased it may be.

The February 2010 report, Analyzing the Accuracy and Time Costs of Web
Application Security Scanners, once again generated heated discussion about web application security scanners. (A similar report was previously published in October 2007.) The new report addressed some criticisms of the 2007 version and included more scanners and more transparent targets. The 2010 version, along with some strong reactions to it, engender some additional questions on the topic of scanners in general:

How much should the ability of the user affect the accuracy of a scan?

Set aside basic requirements to know what a link is or whether a site requires credentials to be scanned in a useful manner. Should a scan result be significantly more accurate or comprehensive for a user who has several years of web security experience than for someone who just picked up a book in order to have spare paper for origami practice?

I’ll quickly concede the importance of in-depth manual security testing of web applications as well as the fact that it cannot be replaced by automated scanning. (That is, in-depth manual testing can’t be replaced; unsophisticated tests or inexperienced testers are another matter.) Tools that aid the manual process have an important place, but how much disparity should there really be between “out of the box” and “well-tuned” scans? The difference should be as little as possible, with clear exceptions for complicated sequences, hidden links, or complex authentication schemes. Tools that require too much time to configure, maintain, and interpret don’t scale well for efforts that require scanning to more than a handful of sites at a time. Tools whose accuracy correlates to the user’s web security knowledge scale at the rate of finding smart users, not at the rate of deploying software.

What’s a representative web application?

A default installation of osCommerce or Amazon? An old version of phpBB or the WoW forums? Web sites have wildly different coding styles, design patterns, underlying technologies, and levels of sophistication. A scanner that works well against a few dozen links might grind to a halt against a few thousand.

Accuracy against a few dozen hand-crafted links doesn’t necessarily scale against more complicated sites. Then there are web sites — in production and earning money no less — with bizarre and inefficient designs such as 60KB .NET VIEWSTATE fields or forms with seven dozen fields. A good test should include observations on a scanner’s performance at both ends of the spectrum.

Isn’t gaming a vendor-created web site redundant?

A post on the Accunetix blog accuses NTO of gaming the Accunetix test site based on a Referer field from web server log entries. First, there’s no indication that the particular scan cited was the one used in the comparison; the accusation has very flimsy support. Second, vendor-developed test sites are designed for the very purpose of showing off the web scanner. It’s a fair assumption that Accunetix created their test sites to highlight their scanner in the most positive manner possible, just as HP, IBM, Cenzic, and other web scanners would (or should) do for their own products. There’s nothing wrong with ensuring a scanner — the vendor’s or any other’s — performs most effectively against a web site offered for no other purpose than to show off the scanner’s capabilities.

This point really highlights one of the drawbacks of using vendor-oriented sites for comparison. Your web site probably doesn’t have the contrived HTML, forms, and vulnerabilities of a vendor-created intentionally-vulnerable site. Nor is it necessarily helpful that a scanner proves it can find vulnerabilities in a well-controlled scenario. Vendor sites help demonstrate the scanner, they provide a point of reference for discussing capabilities with potential customers, and they support marketing efforts. You probably care how the scanner fares against your site, not the vendor’s.

What about the time cost of scaling scans?

The report included a metric that attempted to demonstrate the investment of resources necessary to train a scanner. This is useful for users who need tools to aid in manual security testing or users who have only a few web sites to evaluate.

Yet what about environments where there are dozens, hundreds, or — yes, it’s possible — thousands of web sites to secure within an organization? The very requirement of training a scanner to deal with authentication or crawling works against running scans at a large scale. This is why point-and-shoot comparison should be a valid metric. (In opposition to at least one other opinion.)

Scaling scans don’t just require time to train a tool. It also requires hardware resources to manage configurations, run scans, and centralize reporting. This is a point where Software as a Service begins to seriously outpace other solutions.

Where’s the analysis of normalized data?

I mentioned previously that point-and-shoot should be one component of scanner comparison, but it shouldn’t be the only point — especially for tools intended to provide some degree of customization, whether it simply be authenticating to the web application or something more complex.

Data should be normalized not only within vulnerabilities (e.g. comparing reflected and persistent XSS separately, comparing error-based SQL injection separately from inference-based detections), but also within the type of scan. Results without authentication shouldn’t be compared to results with authentication. Other steps would be to compare the false positive/negative rates for tests scanners actual perform rather than checks a tool doesn’t perform. It’s important to note where a tools does or does not perform a check versus other scanners, but not performing a check has a different reflection on accuracy versus performing a check that still doesn’t identify a vulnerability.

What’s really going on here?

Designing a web application scanner is easy, implementing one is hard. Web security has complex problems, many of which have different levels of importance, relevance, and even nomenclature. The OWASP Top 10 project continues to refine its list by bringing more coherence to the difference between attacks, countermeasures, and vulnerabilities. The WASC-TC aims for a more comprehensive list defined by attacks and weaknesses. Contrasting the two approaches highlights different methodologies for testing web sites and evaluating their security.

So, if performing a comprehensive security review of a web site is already hard, then it’s likely to have a transitive effect on comparing scanners. Comparisons are useful and provide a service to potential customers, who want to find the best scanner for their environment, and useful to vendors, who want to create the best scanner for any environment. The report demonstrates areas not only where scanners need to improve, but where evaluation methodologies need to improve. Over time both of these aspects should evolve in a positive direction.

(UFO label = Unabashed Flamebait Observations)