Training Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging Face

The speakers introduce "Masov," a rigorous benchmark evaluating AI models' ability to reason through complex, chained logic vulnerabilities in access control, arguing that open-source models are essential for defending against rapidly evolving, multi-target cyber attacks 2:15.

Key Takeaways:
• The benchmark tests models in a black-box environment where they must exploit logic-based vulnerabilities across chained microservices without code access, requiring dynamic world modeling to navigate state changes 7:30.
• Current state-of-the-art models struggle with logical leaps; only GPT-5.5 has successfully solved the benchmark, highlighting a critical gap in reasoning capabilities for complex exploitation chains 12:25.
• Traditional defensive stacks are failing because attackers can now target multiple systems simultaneously, necessitating faster, model-driven defense mechanisms that operate at scale without heavy human intervention 5:20.
• Success requires high-quality, human-curated data to teach models dynamic reasoning rather than simple pattern matching, derived from real zero-day vulnerabilities in open-source software 8:30.
• The future of cybersecurity relies on a collaborative ecosystem of strong open-source models to ensure defensive speed and capability outpace attackers through shared post-training data and collaboration 14:50.

The presentation underscores a critical shift in cybersecurity economics, urging the community to leverage open-source AI to rebuild defensive stacks capable of operating at the speed of modern, automated attacks.

Sources:

• 2:15 Introduction to the bench