## Overview This case study is derived from an in-depth technical discussion among members of the Cursor team, a company building AI-powered coding tools. The conversation centers on the application of reinforcement learning (RL) to code generation agents and the unique operational challenges this presents compared to other domains like mathematics or general writing. The discussion is notable for its candid exploration of both theoretical considerations and practical implementation challenges, offering valuable insights into the state-of-the-art of LLMOps for coding agents. Cursor occupies an interesting position in the AI ecosystem: they serve as the interface between large language models and real-world coding tasks, giving them unique access to user feedback and the ability to observe how models perform in production environments. This positioning informs much of their thinking about how to train and deploy coding agents effectively. ## The Unique Challenges of RL for Coding The team identifies several key differences between applying RL to coding versus other domains: **Action Space Complexity**: Unlike math problems where the reasoning process leads to a relatively short answer, coding has a much larger action space. The reasoning in coding is often embedded within the answer itself—the code you write is both the thought process and the solution. This means traditional approaches that work well for math (extended reasoning to arrive at a concise answer) don't transfer directly. **Multi-Step Tool Calling**: Coding agents don't simply generate tokens and receive a reward. Instead, they generate tokens, call tools (like terminals, linters, or file systems), receive responses from those tools, and potentially repeat this process multiple times before reaching a conclusion. This creates an RL problem with a fundamentally different shape—the optimization must account for this iterative, tool-augmented generation process. **Reward Signal Sparsity**: The team discusses the challenge of sparse rewards, particularly for complex tasks like full pull requests. If a model only succeeds one in a thousand times at a complete task, the sparse reward becomes a significant training challenge. They suggest breaking tasks into smaller, more tractable components to reduce this sparsity, though this adds its own complexity. ## Reward Signals and Their Trade-offs The discussion extensively covers different approaches to reward signals: **Test-Based Rewards**: Tests provide something close to ground truth—does this code work or not? However, the team acknowledges limitations: tests can be gamed, lack of coverage allows models to pass tests without truly solving the problem, and test passing doesn't capture code quality or elegance. One participant notes that even passing tests doesn't tell you what the model actually did to pass—it could have made completely unrelated changes that happen to satisfy the test conditions. **Human Feedback**: The team is in a unique position to collect real-world user feedback through the Cursor interface. They discuss various signals: whether users accept edits, whether code "sticks around" after acceptance, whether users switch to different models, and even whether users churn from the product entirely. There's an interesting acknowledgment that thumbs up/thumbs down feedback (as referenced in OpenAI's sycophancy problem) can be problematic—it biases toward users who engage with such mechanisms and may not represent true task success. **Reward Models**: The conversation explores using reward models, particularly those that can see ground truth (what the user actually did after receiving a suggestion). Such models could potentially allow much longer optimization without reward hacking, since they're grounded in reality rather than learned approximations. The team expresses enthusiasm about their position having "basically infinite human feedback" to retrain reward models frequently. **Process vs. Outcome Rewards**: The team discusses why process reward models (PRMs) have fallen out of favor. The issue is that models aren't accurate enough at providing intermediate step scores, and once you apply optimization pressure against such a verifier, you quickly hit limits. Ground truth outcome signals (like math answers or test results) allow for much deeper optimization—the DeepSeek R1 model reportedly used 10,000 RL steps, compared to typical RLHF pipelines that only do about 100. ## Tool Design and Optimization The discussion covers interesting ground on what tools coding agents should have access to: **Terminal Dominance**: Models like o3 are heavily optimized for terminal usage because it's simple and universal—you don't need sophisticated harnesses, just shell access. However, this simplicity comes at the cost of potentially better, more specialized tools. **Language Server Integration**: Linter errors provide significant signal, but require running language servers, which is technically challenging for arbitrary codebases. Cursor has an advantage here because their product comes with extensions that have language servers pre-installed. **Semantic Search**: While grep with enough hops could find anything, semantic search gets there faster, using less context window and compute. **Thinking Tools**: An interesting suggestion is adding explicit "thinking tools" that the model can invoke when it recognizes a task requires reasoning, rather than forcing thinking before any information gathering. This addresses a problem where reasoning models think extensively before even seeing relevant context. **Memory Tools**: The conversation explores the challenge of memory tools—storing information for retrieval in future trajectories. The problem is that storing a memory creates dependencies across trajectories that are difficult to train with standard RL. The team discusses using rule-based or heuristic approaches instead of learned mechanisms, evaluating different memory strategies on benchmarks. **PR/Context Tools**: An interesting proposal is giving agents access to historical PRs and what colleagues have been doing, framing the model as a "capable engineer perpetually on their third day on the job" who needs to understand the codebase's evolution. ## Long Context and Attention Mechanisms The team devotes significant discussion to long context handling: **Cost vs. Capability Trade-offs**: While attention handles long contexts well, costs scale problematically. The team is interested in caching strategies and efficient reuse of context across prompts, particularly relevant with newer, more capable (and expensive) models. **NSA (Native Sparse Attention)**: The DeepSeek attention mechanism is discussed approvingly. It splits attention into three parts: sliding window for recent context, and a two-phase mechanism where the model first attends to compressed block representations to identify the top-k relevant blocks, then does full attention only on those selected blocks. This creates efficient retrieval across long contexts. **Document-Level Attention ("Squid Attention")**: Cursor's internal approach involves independently attending to each document before combining them at the end. This enables caching documents independently and swapping them in at inference time without repaying prefill costs—valuable for features like code tab completion and semantic search in agents. **Hardware Implications**: The team discusses how new GPU architectures (like NVLink-connected GB200 systems with 72 GPUs) enable better long context handling through tensor parallelism that extends beyond the typical 8-GPU mesh, and unified CPU-GPU memory for KV cache storage. ## RL Infrastructure Considerations The infrastructure discussion reveals several important LLMOps considerations: **Complexity Over Standard Training**: RL infrastructure builds on training infrastructure but adds inference components. Both forward/backward passes and rollout generation must be optimized, but for different metrics—inference cares about throughput, not latency, when doing RL. **Prompt Efficiency for GRPO**: With agents, prompts can be massive. The team describes optimizations where they overlap computation: starting to compute prompt KVs while the inference server works on rollouts, then reusing those KVs during backward passes. This avoids repeatedly computing through shared prompts across multiple rollouts. **Weight Synchronization**: Moving parameters between training and inference nodes is a major challenge. Some approaches use asynchronous updates where the inference model is one step behind, generating rollouts with stale weights while training proceeds—this increases training speed but introduces staleness. **Throughput vs. Latency Optimization**: For RL, unlike user-facing inference, you want maximum tokens sampled per GPU rather than minimum latency per token. Techniques like prefill-decode disaggregation become important: prefill once on one prompt, then spin up many decoder workers for parallel rollouts. **Online RL from Production**: For high-volume products like tab completion, it may be possible to use actual production inference as RL data, collecting rewards from user acceptance behavior without needing separate rollout infrastructure. This requires very fast policy deployment cycles but ensures policy-trajectory alignment. ## GRPO vs. PPO Trade-offs The team discusses why GRPO has become popular: **Memory Efficiency**: GRPO doesn't require storing value function weights, reducing memory requirements. **Accuracy Issues with Value Functions**: For hard tasks like math and code, value functions aren't very accurate anyway—the model struggles to predict intermediate values. Multiple rollouts with ground truth rewards provide better value estimates despite higher compute costs. **Historical Context**: The team notes that GRPO existed well before the DeepSeek R1 paper that popularized it, and similar ideas (like REINFORCE with leave-one-out baselines) go back to 2019. ## Future Directions The conversation concludes with speculation about the future of coding agents: **Increased Token Usage**: Models will likely use many more output tokens, with extended tool-calling sequences. However, the team sees waste in current approaches where thinking is discarded after each interaction. **Amortized Reasoning**: Future systems should be able to reuse reasoning across sessions, building up codebase understanding in the background that can be rapidly applied when questions arise. **Codebase Specialization**: Long context or some form of codebase-specific adaptation will be crucial for efficiency, reducing the output tokens needed for each response. **Data Efficiency vs. Compute Efficiency**: As compute becomes more available relative to high-quality data, compute-expensive methods like extensive sampling may become more acceptable. The team observes that the highest quality training data is becoming scarce relative to available compute. This discussion provides valuable insights into the current state of LLMOps for coding agents, highlighting both the unique challenges of this domain and the innovative solutions being developed by practitioners at the frontier.