This case study provides an in-depth look at Cursor's approach to implementing reinforcement learning for coding models and agent systems in production. Cursor is a code editing platform that integrates AI assistance directly into the development workflow, and this transcript reveals extensive technical discussions about their LLMOps infrastructure and methodologies. The conversation begins by establishing fundamental differences between applying RL to coding versus other domains like mathematics. Unlike math problems where reasoning helps arrive at short, verifiable answers, coding involves much larger action spaces and the reasoning is often embedded within the solution itself. The coding domain requires multi-step tool calling processes where models must generate tokens, call various tools, receive responses, and iterate through this cycle multiple times before completion. This creates a significantly different RL optimization landscape compared to simpler generate-reason-answer patterns. One of the core challenges discussed is reward signal design. While mathematical problems often have clear ground truth answers that can be verified, coding tasks present more complex evaluation scenarios. The team extensively discusses test-based rewards, acknowledging both their strengths and limitations. Tests provide near-ground-truth signals when coverage is comprehensive, but they can also lead to gaming behaviors where models find ways to pass tests without actually solving the intended problems. The sparsity of rewards is another significant challenge - models might only pass tests in one out of a thousand attempts, making the training computationally expensive and requiring sophisticated infrastructure to handle the volume of rollouts needed. The infrastructure requirements for RL in coding are substantially more complex than traditional supervised fine-tuning. The system must handle both forward and backward passes for training while simultaneously managing high-throughput inference for generating rollouts. The team describes implementing optimizations where they overlap inference serving with training by pre-computing and caching key-value pairs for prompts, allowing them to reuse computation across multiple rollouts that share the same prompt. This is particularly critical for their use case where prompts can be massive due to large codebases being included in context. Cursor has developed several innovative attention mechanisms to handle long context efficiently. They discuss "squid attention" - a document-level attention mechanism where each document (like different files in a codebase) can be independently attended to and then combined at the end. This allows for caching keys and values for individual documents and swapping them in and out at inference time without recomputing prefill costs. They also explore various sparse attention mechanisms like NSA (from DeepSeek) that use top-k selection to attend to the most relevant chunks of context. The team places significant emphasis on tools and their integration into the RL training process. Beyond basic terminal access, they discuss more sophisticated tools like linter errors, semantic search, and language servers that provide richer signal for training. They've found that terminal access is popular due to its simplicity - it requires minimal harness development while allowing agents to perform virtually any action. However, they're exploring higher-quality tools that could provide better training signal, even if they require more complex infrastructure. Memory systems represent another frontier they're investigating. The challenge with memory tools is the delayed reward problem - storing a memory in one trajectory only provides value in future, unrelated trajectories. This creates significant credit assignment challenges during training, as you need to run multiple rollouts across different contexts to get meaningful reward signals for memory storage actions. They're experimenting with heuristic-based approaches rather than pure RL for memory generation and retrieval. The conversation reveals sophisticated thinking about reward model development. Rather than relying solely on thumbs up/down feedback (which they note caused issues for other systems), they're exploring using actual user behavior as reward signals. They can observe whether users accept suggested edits, whether they switch away from their model, and ultimately whether users continue using their service. This provides more authentic feedback than explicit rating systems, though it requires careful infrastructure to capture and process these signals. For algorithm selection, they discuss the trade-offs between GRPO (Group Relative Policy Optimization) and PPO (Proximal Policy Optimization). GRPO requires more compute due to multiple rollouts per prompt but eliminates the need for value functions, which they've found to be inaccurate for complex coding tasks. The high variance typically associated with GRPO can be managed with large enough batch sizes, which their infrastructure can support. The team anticipates significant changes in how coding agents will operate in the future. They expect models to use far more tokens, both in input (longer context windows) and output (longer reasoning sequences). However, they're concerned about efficiency and believe future systems should be able to amortize computational work across interactions rather than redoing analysis for each request. They envision systems that can build and maintain understanding of codebases over time, potentially through background processing that creates reusable knowledge representations. The discussion also touches on hardware considerations, particularly how new GPU architectures like GB200 and NVL72 enable longer context processing through improved tensor parallelism and unified memory architectures. These advances make it feasible to store key-value caches across much larger context windows, though they note that the fundamental quadratic attention cost still limits scalability. Throughout the conversation, there's a strong emphasis on bridging the gap between research benchmarks and real-world usage. While much RL work focuses on passing test suites like SWE-bench, they're more interested in optimizing for actual developer workflows - tasks like adding console logs across files, improving code quality, and handling the subjective aspects of code style and elegance. This requires developing evaluation metrics that capture human preferences and real-world utility rather than just functional correctness. The case study demonstrates the complexity of deploying RL for coding at scale, involving considerations spanning algorithm design, infrastructure optimization, reward signal development, and user experience. Cursor's approach represents a comprehensive LLMOps implementation that addresses the full stack of challenges in bringing RL-trained coding models to production environments.