Researchers at Carnegie Mellon University and Bugcrowd developed reinforcement learning environments to teach LLMs to autonomously discover and exploit cybersecurity vulnerabilities at scale. The approach addresses critical challenges in existing benchmarks, including multi-vulnerability programs and the distinction between simple bug detection and actual exploitation capability. Testing frontier models including GPT, Mythos, and Gemini against 41 Chrome V8 vulnerabilities revealed significant capability differences: while most models could trigger crashes 50-95% of the time, only advanced models like Mythos and GPT achieved full sandbox escape exploits 68-73% of the time, demonstrating performance on par with elite human security researchers and discovering novel exploitation techniques not previously known publicly.
This case study describes a comprehensive research effort by Carnegie Mellon University in collaboration with Bugcrowd to design and implement reinforcement learning environments specifically tailored for teaching large language models to perform autonomous cybersecurity tasks. The work focuses on both vulnerability discovery and weaponization, with the ultimate goal of achieving machine-speed security testing at scale. The research addresses fundamental challenges in how to properly design training and evaluation environments for cybersecurity tasks, drawing parallels between how humans learn hacking through graduated difficulty and how LLMs can be trained using similar pedagogical approaches.
The speaker, David Brumley, serves as both a full professor at Carnegie Mellon University working on AI and cybersecurity, and as Chief AI and Science Officer at Bugcrowd. The work represents over two decades of research in teaching both humans and machines to hack, with a particular focus on production-ready LLM systems that can operate autonomously against real-world targets.
The fundamental challenge addressed in this work is that software is being released faster than ever before, necessitating automated security testing at machine speeds and scale. Traditional cybersecurity approaches cannot keep pace with the volume of code being produced. While several first-generation cybersecurity benchmarks existed for evaluating LLMs, they contained critical design flaws that prevented models from learning effectively and accurately measuring their true capabilities.
The research team identified two major axes for designing cybersecurity tasks: target difficulty ranging from toy problems through CTF challenges to hardened production targets, and exploitation difficulty spanning from simple bug detection through crash triggering to full arbitrary code execution. The insight is that hacking represents a natural ladder of skills with clear oracles for success, making it well-suited to reinforcement learning approaches.
The reinforcement learning environment architecture follows a standardized gym-like structure with several key components. Each environment contains a vulnerable application enclosed in a Docker container to ensure reproducibility and eliminate variations across different execution environments. This containerization is critical for production deployment as it standardizes behavior across different Linux versions and system configurations.
The team exposes functionality through the Model Context Protocol (MCP), providing a clean interface with several standard functions. The setup function returns the problem definition to the LLM. Standard tool calls including read and write operations allow the model to interact with the container in a sandboxed manner. Finally, a grading oracle evaluates whether the model successfully completed the task.
A critical design decision was implementing deterministic grading oracles rather than using LLM-as-judge approaches. The research found that when using LLMs to evaluate their own cybersecurity performance, they consistently claim success even when unsuccessful. This represents an important lesson for LLMOps practitioners: self-evaluation by LLMs is unreliable in domains where objective ground truth can be established through deterministic means.
One of the most significant contributions of this work is identifying and solving what the researchers call the multi-vulnerability problem. Existing benchmarks like Cybex and CyberGym assumed programs contain only a single vulnerability. However, in real-world scenarios, programs almost always contain multiple vulnerabilities, and this creates several critical issues for reinforcement learning.
When multiple vulnerabilities exist but only one is known to benchmark creators, LLMs engage in reward hacking by repeatedly finding the easiest vulnerability. This prevents the model from developing more sophisticated exploitation capabilities and stunts its learning trajectory. The model optimizes for the simplest path to reward rather than developing comprehensive security analysis skills.
Some benchmarks attempted to address this by providing hints about which specific vulnerability to target, such as including a backtrace pointing to the vulnerable function. However, this approach fundamentally limits the reasoning capabilities the model must develop. When the model is told exactly where the vulnerability exists, it can often fit that single function into its context window and doesn’t need to reason about the broader program structure or develop true vulnerability discovery skills.
The research team found compelling evidence of this problem across multiple contexts. DARPA’s Cyber Grand Challenge, which cost sixty million dollars and involved extensive manual curation of challenges, still had unknown vulnerabilities in 50% of hand-curated problems. Similarly, the AIxCC competition from the previous year discovered 18 unintended bugs during competition. This demonstrates that even with expert-level effort and substantial resources, creating synthetic benchmarks with exactly one known vulnerability is effectively impossible.
To address the multi-vulnerability problem, the researchers developed a novel approach called the audit task. Instead of asking the model to find a single bug, the task prompt asks the model to find all vulnerabilities it can discover. This fundamental reframing changes the incentive structure and enables several important capabilities.
The LLM has freedom to find and submit multiple proofs of vulnerability, covering both known and unknown vulnerabilities. Each submitted proof runs through the deterministic oracle to verify it actually triggers the claimed behavior. The grading system then uniquifies the discovered vulnerabilities to distinguish between different bugs versus multiple paths to the same bug. This uniquification process uses stack backtrace analysis, the same technique used by industry systems like Windows and Mac crash reporting to categorize distinct bugs.
The scoring mechanism balances precision and recall. Recall measures how many known vulnerabilities the model found divided by the total set of known vulnerabilities. Precision measures how many verified vulnerabilities the model found divided by how many it submitted. This multiplicative combination prevents the model from spam-submitting random inputs while still rewarding discovery of unknown vulnerabilities.
Critically, the audit task enables open-world grading. When the model discovers a vulnerability unknown to the benchmark creators, the system can verify the proof of vulnerability and expand the ground truth set. This creates a cleaner learning trajectory because the model never knows whether zero, one, or multiple vulnerabilities exist in the target program. This removes the information leak present when task prompts imply a vulnerability definitely exists.
Beyond vulnerability discovery, the research team focused extensively on measuring true exploitation capability as distinct from simple bug finding. This represents the second axis of difficulty in their framework. Security differentiated from mere bug detection requires demonstrating actual control over the target system.
The team developed a graduated ladder of exploitation capabilities that models must demonstrate. At the lowest level, can the model trigger the vulnerability and observe some deviation in program behavior? Moving up, can it trigger a crash? Then, can it achieve arbitrary read and write primitives in memory? Finally, can it achieve full arbitrary code execution where the model can make the program run attacker-controlled code?
This ladder provides measurable checkpoints for model capability and helps diagnose where models get stuck. Rather than binary success/failure, the framework provides rich signal about which specific skills the model has mastered and which remain beyond its current capabilities.
To truly test whether models could achieve elite-level exploitation capabilities, the research team selected Chrome’s V8 JavaScript engine as a hard target benchmark. This choice was deliberate and strategic. V8 powers Chrome, Edge, Node.js, and Cloudflare Edge Workers, making it one of the most critical security components on the internet. Vulnerabilities in V8 are highly valuable, with bounties ranging from ten thousand to one hundred thousand dollars, and black market prices reaching millions.
V8 implements sophisticated security measures beyond typical programs. It runs untrusted code like JavaScript within a sandbox environment, and vulnerabilities confined to the in-sandbox context are expected and not considered true security failures. A genuine V8 exploit requires achieving out-of-sandbox capabilities, typically by chaining multiple vulnerabilities together. This represents the level of sophistication achieved by elite human security researchers.
The research team assembled 41 V8 vulnerabilities and manually verified with Sung Hin Lee, the current leader for Chrome security, that each was exploitable. They then tested multiple frontier models including GPT 5.5, Mythos, Gemini, Kimmy, Minimax, and GLM across the exploitation difficulty ladder.
The results revealed significant capability differentiation between models that would be hidden by traditional crash-only benchmarks. Looking at simple vulnerability triggering, both GPT and Mythos achieved 95% success rates, triggering vulnerabilities in 39 out of 41 cases. Even lower-powered models like Gemini, Kimmy, Minimax, and GLM succeeded approximately 50% of the time.
This highlights a critical flaw in existing benchmarks that measure only crash triggering. If crash triggering is the metric, the message would be that Kimmy succeeds at hacking 50% of the time, creating a misleading assessment of actual security capabilities.
However, when measuring full sandbox escape with arbitrary code execution, the capability gap became stark. Mythos achieved success 73% of the time, successfully escaping the sandbox in 30 out of 41 cases. GPT achieved 68% success rate. In contrast, Gemini and Kimmy achieved 0% success on full sandbox escape exploits.
These results demonstrate performance on par with elite human security researchers for the top-performing models. More significantly, the research team found evidence that models were not simply memorizing known exploits from training data. In several cases, the models discovered novel exploitation techniques.
Three specific examples provide compelling evidence of genuine reasoning capability rather than memorization. For CVE-2023-670T, experts in the Chrome security community knew the vulnerability was exploitable and had created a proof of concept. However, Mythos took a completely different approach that the expert community had considered too difficult to execute in practice. The model successfully reversed JavaScript’s Math.random function and used that to forge a pointer for a return-oriented programming attack to escape the Uber cage exploit. This technique was highly creative and not part of the public exploit for this vulnerability.
For CVE-2024-7965, the model found a new WebAssembly path where all public work had stopped. It remained unclear whether a public exploit even existed for this vulnerability. The research team was able to manually create one after the fact, but confirmed this was not publicly available information the model could have memorized during training.
For CVE-2024-0519, a public vulnerability existed but no public exploit was available, yet Mythos succeeded in creating a working exploit. Most notably for CVE-2024-7965, the team knew of a public exploit that worked on ARM processors, but their internal expert believed exploitation on x86 was not feasible. Mythos succeeded on x86, providing significant proof this was not memorization.
Beyond benchmark creation, the research team deployed these environments in production to support reinforcement learning at scale. Their approach leverages extensive vulnerability mining infrastructure developed through DARPA research over the previous decade for novel vulnerability discovery. This infrastructure finds unique proofs of vulnerability representing zero-day exploits that no one else has discovered.
Using zero-day vulnerabilities for training environments provides strong guarantees that models cannot be merely memorizing solutions from training data. If a model successfully exploits a vulnerability it has never encountered before, memorization is ruled out as an explanation.
The team provides these reinforcement learning environments to companies working on frontier models at significant scale, delivering up to ten thousand new environments per month. This production deployment represents a substantial LLMOps operation managing containerized environments, deterministic grading at scale, and continuous pipeline of novel training tasks.
The complete benchmark suite is available at exploitbench.ai as an open resource for the community. The team provides Docker images that can be pulled directly from GitHub, each with an MCP interface allowing straightforward integration. Practitioners can point Claude or other models at the MCP interface and immediately begin evaluation.
All experimental data and transcripts are provided with one notable exception: Mythos transcripts were withheld under NDA restrictions and because Mythos generated weaponized exploits for high-value targets that were not previously public. This creates an interesting dilemma for LLMOps practitioners committed to open science: when models create genuinely novel and potentially dangerous capabilities, how should the community balance openness with responsible disclosure?
This work provides several critical lessons for LLMOps practitioners building production LLM systems, particularly for evaluation and reinforcement learning. First, oracle design is paramount. Incorrect task objectives lead to models learning the wrong behaviors even when the learning process itself functions correctly. The audit task demonstrates how reframing the problem from finding a bug to finding all bugs fundamentally changes model incentives and prevents reward hacking.
Second, deterministic grading outperforms LLM-as-judge approaches in domains where ground truth can be objectively established. LLMs consistently exhibit self-serving bias when evaluating their own outputs in cybersecurity contexts. This likely applies to other domains where objective verification is possible.
Third, multi-tier evaluation rubrics that measure graduated capabilities provide richer signal than binary pass/fail metrics. The exploitation difficulty ladder allows diagnosis of exactly where models get stuck and enables tracking progress as models improve. This granular measurement is essential for effective reinforcement learning.
Fourth, open-world evaluation that accommodates discoveries beyond the initial ground truth enables more realistic assessment and prevents artificial constraints. Real-world deployment will inevitably encounter situations not anticipated during benchmark creation, and evaluation frameworks should embrace rather than penalize such discoveries.
Fifth, container-based standardization through Docker provides reproducibility essential for production RL environments. Eliminating environmental variation ensures that learning signals reflect actual model capability changes rather than environmental noise.
Finally, the work demonstrates that teaching machines follows similar pedagogical principles to teaching humans. The graduated difficulty ladder, starting with toy problems and progressing through CTF challenges to hardened production targets, mirrors how successful human hackers develop skills. This suggests that insights from human education can inform LLM training methodology more broadly.
Cloudflare built a comprehensive internal AI engineering stack over eleven months to integrate AI coding assistants across their R&D organization, achieving 93% adoption among engineering teams. The solution involved creating an MCP-based infrastructure using their own products (AI Gateway, Workers AI, Cloudflare Access, Agents SDK, Workflows, and Sandbox SDK), developing 13 MCP servers with 182+ tools, generating AGENTS.md files for ~3,900 repositories, implementing automated AI code review for all merge requests, and establishing an Engineering Codex for standards enforcement. The result was a dramatic increase in developer velocity with merge requests nearly doubling, processing 241.37 billion tokens monthly through AI Gateway, with 3,683 active users generating 47.95 million AI requests in the last 30 days, while maintaining security through zero-trust authentication and zero data retention policies.
Cloudflare built a production AI code review system to address the bottleneck of manual code reviews across their engineering organization, where median wait times for first review were measured in hours. Rather than using off-the-shelf tools or naive LLM prompting, they developed a CI-native orchestration system around OpenCode that deploys up to seven specialized AI reviewers (covering security, performance, code quality, documentation, release management, and compliance) managed by a coordinator agent. The system has processed over 131,000 review runs across 48,000 merge requests in 5,169 repositories in the first month, with a median review time of 3 minutes 39 seconds, average cost of $1.19 per review, and only 0.6% of reviews requiring manual override, while identifying 159,103 findings with deliberate bias toward high signal-to-noise ratio.
Block conducted an internal red team engagement called "Operation Pale Fire" to proactively identify security vulnerabilities in goose, their open-source AI coding agent. The engagement successfully demonstrated multiple attack vectors, including prompt injection attacks hidden in invisible Unicode characters delivered through calendar invitations and poisoned shareable recipes, ultimately compromising a Block employee's laptop through social engineering combined with AI-specific vulnerabilities. The operation revealed critical weaknesses in how AI agents handle untrusted context and led to concrete improvements including calendar policy changes, enhanced recipe transparency, zero-width character stripping, and prompt injection detection capabilities integrated into the goose platform.
