Teaching AI to Find Real Vulnerabilities — Prof. David Brumley, Bugcrowd

Professor David Brumley discusses designing reinforcement learning environments for cybersecurity, arguing that training AI to hack requires mimicking human learning through a ladder of increasing difficulty 0:20. He highlights that current benchmarks often fail because they assume single vulnerabilities, leading models to "reward hack" the easiest bug rather than developing true exploitation skills 5:45.

Key Takeaways:
• Human hackers learn by studying write-ups and practicing on graduated scales, a method Brumley applies to teaching LLMs 1:50.
• Traditional benchmarks are flawed because they often contain only one known vulnerability, causing models to exploit the easiest bug repeatedly 10:15.
• Brumley proposes an "audit task" where models must find all vulnerabilities, using precision and recall metrics to prevent spamming and reward hacking 13:45.
• Evaluating against hardened targets like Chrome's V8 engine reveals that while models can trigger crashes, few can achieve full arbitrary code execution 18:25.
• Frontier models like Mythos and GPT demonstrate non-memorized skills by discovering novel exploits for known vulnerabilities 22:50.

Brumley concludes that effective cybersecurity AI training relies on deterministic oracles and expert-designed environments that test for genuine capability rather than simple bug detection 26:30.

Sources:

• 0:20 Introduction to teaching computers to hack via reinforcement learning.
• 1:50(https://www.youtube.com/watch?v=ZFxh7sqbUZo?t=110