Posts (page 14 of 43)

• Resolutions for a New Year of Vulns Dec 26, 2017

  Throughout 2017 I explored vuln data to highlight strategies for measuring and maximizing the efficiency of vuln discovery. The primary themes were budget and time — deciding how best to allocate money among different approaches plus evaluating the triggers and frequency of security testing. I was fortunate to present much of this at various conferences, which gave me a chance to collect feedback and engage in interesting discussions about the challenges that DevOps teams face.

  In 2018 I’ll have even more data with more dimensions to explore. While it’s important to lay out a strategy for finding vulns in an app. Knowing what risk an app has is only the start. Reducing that risk is the important next step. Strategies for resolving vulns will be a new theme for the new year.

  Resolving vulns relates to a trope along the lines of, “Show me your budget and I’ll tell you what you value.” This applies to time as well as money. The vulns you choose to fix and the speed with which you fix them reflect an investment in security. Ideally, it also reflects an understanding of risk.

  As a teaser of the content coming next year, here’s a graph that reflects how an organization resolves vulns. It shows the deviation in resolving a specific category of vuln from the average resolution time across all vulns. Fewer days represents a faster fix, greater days a longer one.

  

  In this graph, vulns due to misconfigurations are resolved close to the average, whereas redirects and forwards languish in the DevOps queue for quite a while longer.

  We can follow many paths of discussion from a graph like this. Several things influence the speed of fixes, from complexity of code to risk they carry. In the case of redirects and forwards, it seems fair to guess that such vulns represent little or no risk — a hypothesis further reinforced by the category disappearing from the recent OWASP Top 10 update. (We’ll also examine how prevalence of a vuln type impacts its resolution.)

  I haven’t even noted what the average number of days is. In a way, it doesn’t matter. The vuln category at the top of the graph clearly receives more attention than the two I’ve labeled. In other words, DevOps teams have a clear priority for such vulns. That’s one indicator of the distinction of BugOps vs. DevOps. Even better is if that category were to completely disappear over time.

  It will still be important to discuss threat models and how they help measure risk, especially working with clear, well-considered models instead of falling into counterproductive traps. From there, we’ll take a view of the risk we measure and explore the different ways to reduce it.

  This year has only a few days left. A few days left should be the typical response to fixing vulns. Come back next year to find out what new vuln data says about those few days.

• Secure Design Practices for Verifying Vuln Fixes Dec 12, 2017

  The pen test lifecycle is coming to a close. The previous posts have weighed heavily on getting the process started and running smoothly. After all, it’s important to identify vulns within your apps. But most important is fixing them so the app’s users and data can remain well-protected.

  Part of keeping a pen test successful is ensuring that the pen testers submit findings with clear risk and reproduction steps. This helps developers understand and address the problem.

  A pen test isn’t over once the final report has been delivered. Ideally, the pen testers will remain available to confirm the fixes for each of their findings. Here are a few things to keep in mind when going through the results of a pen test.

  1. Resolve Findings Promptly

  The time it takes to release fixes is affected both by the complexity of the vuln as well as the maturity of the DevOps team. An efficient team can release a fix within a few days. Unfortunately, sometimes a fix can take a few months.

  Part of the process for resolving findings is considering their risk. Critical vulns should be addressed quickly. Other vulns may not need to be addressed at all. These latter types of vulns fall into an “Accepted Risk” category.

  Accepted Risk isn’t a loophole to avoid fixing issues. It’s an acknowledgement that some design decisions trade off different levels of security. Some decisions may be choose between two competing approaches to usability without sacrificing security significantly for either one.

  2. Address the Underlying Cause

  One reason to have pen testers confirm a fix is to ensure the vuln is correctly resolved. In the worst case, a developer may have simply put an attack payload on a deny list instead of applying a proper countermeasure. As an egregious example, this would be like blocking “alert” or “javascript” to prevent a cross-site scripting vuln instead of correctly applying the relevant output encoding for the payload.

  3. Look for Vuln Patterns Elsewhere

  The nature of a blackbox pen test means that the pen testers don’t have full insight or visibility into the app. As a developer, you do.

  Certain types of vulns have patterns that lends themselves to quick searches for similar problems elsewhere in the code. This might be a misused function that lead to an injection attack, a missing CSRF token, or a missing access control check. Take the time to review similar code paths for this kind of vuln.

  Conduct a postmortem for high and critical vulns. This exercise brings a team together to discuss what went wrong, whether a proposed fix will be effective, and create action items for additional countermeasures. The goal of a postmortem is to collect and share knowledge about important security vulns. It’s not an exercise in placing blame. While the presence of a vuln implies that someone did make a mistake, it’s important to understand how the mistake was made and how it could be prevented in the future.

  4. Create a Regression Test

  When possible, create a regression test that can reproduce and catch the vuln in the future. A regression test is also a way to discover other areas where a vuln may be lurking. (This ties into step three above.) Tests should be an integral part of a DevOps pipeline. They ensure code quality as well as build confidence in an app’s stability under constant change.

  Always prefer creating a regression test over making a comment in the code. Comments go stale quickly, may be ambiguous, and may often be incorrect. Code that is self-documenting (e.g. informative function and variable names) and whose functions are short enough to fit within a page of text is far easier to maintain.

  Going through the effort of creating regression tests also helps reinforce an understanding of the nature of a vuln. By reproducing it, a developer can get a better appreciation for the underlying problem and consider more effective countermeasures. It’s also a way of determining whether your test infrastructure is sufficiently flexible enough to handle certain types of error cases. For example, if you have no way of rendering a web page or inspecting the DOM produced by a resource call, then you may not be able to effectively test for cross-site scripting.

  It may not be possible to create a regression test for every vuln, especially complex ones that require multiple steps or exercise a series of flaws.

  A pen test should equip you with the knowledge about the current risk exposed by your app and provide recommendations for reducing the risk of the app’s data or users being compromised. The pen test may be a point in time snapshot of the app’s flaws, but how you address the vulns will have a lasting improvement to its security.

  Make it harder for vulns to appear. Refactor code so that developers can more easily do the right thing by default. Check out this blog post for additional ideas on making vulns a rare occurrence.

  Now that you’ve completed a pen test, it’s time to start planning for the next one. The frequency of testing can align with a calendar, e.g. semi-annual or quarterly, or align with major code releases. In any case, the speed with which you’re able to resolve vulns will also be a signal for how well you’ll be able to manage more testing.

• Avoid BugOps, Do DevOps Oct 26, 2017

  DevOps aims to release code quickly with confidence. Frequent, fast releases aren’t the hard part. The challenge is achieving justifiable confidence that changes won’t break the production environment and, when that inevitably happens, that teams are able to quickly analyze and resolve problems.

  

  This level of maturity requires smart investment in automation, testing, and monitoring. (And people! We’ll dive into that angle in a different post.) Automation increases the pace at which code goes through review, testing, and deployment. Testing (hopefully) detects errors and halts the deployment to prevent bugs from reaching production. Monitoring helps ensure the app remains stable under constant change by providing feedback about its health and activity.

  Code isn’t perfect. Perhaps an automated step mishandled an error condition, or testing had a gap in coverage that didn’t exercise functionality affected by a code change. Or maybe the app’s monitoring missed a type of event or omitted useful info. Bugs happen.

  Vulnerabilities are bugs that impact the security of the app, its data, or its users. Vuln discovery is important. It’s one of the reasons bug bounties have become so pervasive. We want to know what bugs our apps have, especially if they’ve reached production. And we also have to fix them.

  Being able to fix vulns fast is commendable. But a too-narrow focus on speed can turn DevOps into BugOps. BugOps is releasing code quickly to fix vulns without considering their underlying cause. It leads to an endless loop of find-fix-repeat. While it’s important to fix vulns promptly, just adding quick patch after quick patch only makes an app more brittle.

  Metrics are a critical tool for decision making. But a shortsighted devotion to metrics aggravates the BugOps mentality. Adhering to SLAs while ignoring root causes creates the illusion of secure code. A quickly-patched app may still rest on a weak architecture.

  Another tenet of DevOps is building feedback loops — collecting and responding to actions and events throughout the development pipeline. This should apply equally to vuln discovery.

  When vulns appear in production, it’s especially important to analyze how they arrived there and what quality controls they bypassed. They might be due to a mistake, where a coding guideline or established process wasn’t followed. Or they might be due to a misunderstanding, where some flaw in the app’s architecture was exposed or some process didn’t exist.

  This analysis can inspire fundamental changes to an app’s design that sweep away whole classes of vulns. Or it may introduce controls that make the exploitation of vulns less impactful and more evident.

  Good analysis provides insight into gaps in tools, knowledge, or process. For example, if your testing framework can’t model the types of vulns that are being reported, then you have two problems. One, you won’t be able to create effective regression tests. Two, you’re being underserved by automation.

  Good metrics provide insight into how well a DevOps team handles security. Collecting metrics emphasizes what topics are important (hence, worthy of measure). Metrics over time produce trends. Trends provide feedback about the effectiveness of security tactics such as introducing a new tool, adjusting a process, or adopting new programming patterns. Some useful metrics related to vuln discovery are

  • Type of discovered vulns. Do certain categories stand out? Do they share similar causes?
  • Risk of discovered vulns. Ideally, this would be a common rating based on severity indicated by a CVSS score. No rating is perfect, but CVSS provides a common frame of reference for severity that informs risk.
  • Speed of fix. What was the time between discovery of the vuln and the code commit that fixed it? How does this measure against expectations or explicit SLAs?
  • Speed of deployment. How long does it take for a commit to reach production? Is there a fast-path for code to address critical issues? Does the app have feature flags that can trivially enable/disable problem areas until a fix is ready?
  • Location (e.g. files, objects, functions) of vulns within source code. Review the commit history associated with vulns. Do developers repeatedly address vulns in a particular code path? Is a vulnerable pattern repeated elsewhere, waiting to be reported as vulnerable? Are particular developers responsible for weaker code? Is any automation or tool capable of identifying the vuln?
  • Staleness of the location of vulns within source code. In addition to space (i.e. where), capture the time associated with the vuln’s fix. When was the last time the affected code was touched? Is it related to older, legacy code? Is it in newer code? This can help highlight whether the app is on a path of cleanup to improve its overall quality or remains stuck with the same eternal programming mistakes.
  • Effort to fix. Related to speed, this is more about the cost associated with fixing vulns. It may be a measure of hours required to analyze the vuln and commit a fix. It could also be the number of people involved in the process. For example, a vuln might require a complex fix or many engineering discussions to weigh trade-offs.

  Avoid letting a swarm of vulns chase your team down the BugOps path. As you fix vulns, take the time to figure out how they might have crept into production, adjust tools and processes to catch similar errors that might occur in the future, and track metrics that help show what kind of progress your DevOps team is making to reduce risk.

  Keep an eye out for vulns. Keep your vision on the processes that make DevOps successful.