AI & LLMs – An ASW Topic Recap
LLMs and generative AI were unavoidable appsec topics this year. Here’s a recap of some relevant articles and associated interviews.
Background
- What Is ChatGPT Doing…and Why Does It Work? — Stephen Wolfram Writings
- What is AI? - MIT Technology Review
- Everyone Is Judging AI by These Tests. But Experts Say They’re Close to Meaningless – The Markup
Prompt injection & manipulating models
- ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs – it was fun to see ASCII art appear as an attack vector
- HiddenLayer Research - Prompt Injection Attacks on LLMs – towards a shared language for describing attack techniques and failure modes
- Challenges in Red Teaming AI Systems - Anthropic
- Exploring Large Language Models: Local LLM CTF & Lab - Bishop Fox – have fun with a CTF
- Prompt Airlines – more fun from Wiz
Finding flaws & augmenting appsec
- GitHub - google/oss-fuzz-gen – leveraging LLMs to guide fuzzers. This is probably one of the most appealing and impactful uses I’ve seen
- No, LLM Agents Cannot Autonomously “Hack” Websites – a practitioner’s observations on recent research, plus this follow-up article
- Project Naptime: Evaluating Offensive Security Capabilities of Large Language Models – promises of potential, but remains quite basic
- Using AI for Offensive Security - CSA – rather high level and has more optimism about models actually reasoning (rather than just being really sophisticated non-deterministic pattern matchers)
- DARPA awards $14 million to semifinal winners of AI code review competition
- Deconstructing the AI Cyber Challenge (AIxCC)
Episode 284 (segment 1)
Caleb Sima demystified some of the hype around AI and pointed out how a lot of its security needs match the mundane maintenance of building software. We didn’t get into defining all the different types of AIs, but we did identify the need for more focus on identity and authenticity in a world where LLMs craft user-like content.
Episode 284 (segment 2)
Keith Hoodlet stopped by to talk about his first-place finish in the DoD’s inaugural AI Bias bug bounty program. He showed how manipulating prompts leads to unintentional and undesired outcomes. Keith also explained how he needed to start fresh in terms of techniques since there’s no deep resources on how to conduct these kinds of tests.
Be sure to check these out for my variants on the “walks into a bar” joke.
The AI conversations continued with Sandy Dunn, who shared how the OWASP Top 10 for LLMs came about and how it continues to evolve. We talked about why this Top 10 has a mix of items specific to LLMs and items that are indistinguishable from securing any other type of software. It reinforced a lot of the ideas that we had talked about with Caleb the week before.
Stuart McClure walked through the implications in trusting AI and LLMs to find flaws and fix code. The fixing part is compelling – as long as that fix preserves the app’s intended behavior. He explains how LLMs combined with agents and RAGs have the potential to assist developers in writing secure code.
Allie Mellen pointed out where elements of LLM might help with reporting and summarizing knowledge, but where they also fall short of basic security practices. LLMs won’t magically create an asset inventory, nor will they have context about your environment or your approach to risk. She also notes where AI has been present for years already – we just call it machine learning as applied to things like fraud detection and behavioral analysis.
Subscribe to ASW to find these episodes and more!
-480x320.png)