## Overview Replit, the cloud-based development platform, launched their "Replit Agent" in September 2024 after approximately 15 months of development. The agent is designed to lower the barrier to software creation by allowing users to build web applications from natural language prompts without writing code. This case study, derived from a talk by Michele Catasta (who leads Replit's AI team), provides extensive insights into the LLMOps practices and architectural decisions behind building and operating a complex agentic system in production at scale. The core philosophy behind Replit Agent differentiates it from other coding agents in the market. Rather than aiming for full autonomy or acting as a "replacement for a junior software engineer," Replit positioned their agent as a product that keeps the user at the center. The agent shows users what it's doing, asks questions, and requests feedback. This design choice directly addresses one of the fundamental challenges in agentic systems: the compounding error problem where agents make mistakes over long trajectories and eventually crash or fail to accomplish goals. ## Multi-Agent Architecture Replit Agent employs a multi-agent architecture, which they arrived at iteratively rather than starting with from the outset. The team began with the simplest possible architecture—a basic ReAct (Reasoning and Acting) loop—and scaled up complexity only when they encountered limitations, specifically when too many tools led to too many errors. The multi-agent system includes several specialized sub-agents: - A **manager agent** that orchestrates the workflow, similar to a manager in a real software team - **Editor agents** that perform code modifications and file operations - A **verifier agent** that can interact with the application, take screenshots, run static checks, and validate that the agent is making progress The key insight behind this architecture is scope isolation. Each sub-agent has the minimum necessary tools and instructions visible to it. The rationale is straightforward: the more you expose to a sub-agent, the more opportunities it has to make incorrect choices. This principle of minimal scope per agent appears to be a critical success factor for maintaining reliability. ## Tool Calling Innovation: Code-Based DSL One of the most technically interesting aspects of Replit Agent is their approach to tool calling. They started with standard function calling APIs from providers like OpenAI and Anthropic, but found these "fairly limited" despite being "elegant and easy to use." The team encountered numerous challenges with function calling reliability, experimenting with tricks like reordering arguments, renaming parameters to guide LLM reasoning, and other prompt engineering techniques. Eventually, they took a "complete detour" and decided to invoke tools through code generation instead. Their approach works as follows: they provide all necessary information to the model in context, allow it to reason through a chain of thought, and then have the model generate a Python DSL (a stripped-down version of Python) that represents the tool invocation. This generated code is parsed on their backend, and if it's compliant with their schema, they know it's a valid tool invocation. If not, they retry. This technique achieved approximately 90% success rate for valid tool calls, which is particularly impressive given that some of Replit's tools are complex with many arguments and settings. The team credits LLMs' strong code generation capabilities as the reason this approach works well—essentially leveraging the models' strengths rather than fighting against function calling limitations. ## Model Selection and Trade-offs Replit primarily relies on Claude 3.5 Sonnet for their main agent operations, describing it as "a step function improvement compared to other models" for code generation and editing tasks. They experimented with fine-tuning frontier models early on but found that the out-of-the-box Sonnet performance was sufficient for their needs. For auxiliary tasks, they use a "long tail" of calls to other models. GPT-4 mini is used for compression and other "watchdog" tasks. They also fine-tune embedding models for file system retrieval. When asked about prioritizing the trade-off triangle of accuracy, cost, and latency, the answer was unequivocal: accuracy first, cost second, latency last. The rationale is that low accuracy is what frustrates users most, and there's an assumption that frontier models will continue to become cheaper, making cost optimization less critical in the long run. ## Error Recovery and Debugging Strategies The agent employs multiple strategies for recovering from errors: **Retry with feedback**: When a tool call fails, the agent retries with the error message passed back. The team notes that "even from that basic self-debugging, it already understands how to make progress." **Reflection**: Every five steps (configurable), the agent stops and reasons about whether it has actually made progress. This serves as a recovery strategy where the agent can roll back to a previous state and restart, adding "enough randomness to the process" to explore different solution paths. **Human-in-the-loop**: As a last resort, the verifier agent can invoke a tool to ask the user for input, particularly when dealing with errors that require human judgment or testing. The "ask human" tool is not just a simple fallback—it's a deliberate design choice. The team found that by building something more engaging and asking users for feedback, they address the compounding error problem more effectively than trying to achieve full autonomy. ## Memory Management and Long Trajectories Trajectories in Replit Agent frequently extend to tens of steps, with some users spending hours on single projects. This creates significant challenges for memory management. The team employs several techniques: - **Truncation criteria** to avoid keeping irrelevant memories - **Memory compression** when jumping between sub-agents - **Further compression** when steps are successfully completed, retaining only high-level descriptions Even with these optimizations, the compound probability problem remains: even with a high probability of success on individual steps, reliability drops quickly after around 50 steps. Their main insight is to "do as much progress as possible in the first few steps"—essentially front-loading the most important work to avoid the pitfalls of very long trajectories. ## Observability with LangSmith The team adopted LangSmith early in development for observability, emphasizing that observability is essential for any serious agent work. As Catasta noted: "Don't start to do any serious agent work without any level of observability... if you don't have a way to observe how your system behaves, you're basically just building a prototype." A particularly innovative technique they use is **trajectory replay**. At every step of the agent, they store the complete state in LangSmith traces. When users report bugs, rather than manually reading through entire traces, they: - Pinpoint which step the issue occurred - Pull the trace via the LangSmith API - Load it as a state checkpoint - Replay the trace on a new codebase with proposed fixes - Verify if the agent behaves differently This fully automated replay system significantly accelerates debugging for complex agentic systems where traditional debugging approaches are "excruciating." ## Evaluation Challenges The team acknowledges that evaluation remains a significant challenge. They experimented with SWE-bench for testing the core ReAct loop but found it insufficient for end-to-end testing of the complete Replit Agent experience. Honestly, the team admits to launching "based on vibes"—using the product extensively internally and making clear to users that it was an "early access" product. They express hope that academia or industry will develop better public benchmarks for coding agents. Their verifier sub-agent is positioned as a potential solution, with plans to make it more powerful and capable of mimicking user actions on the artifacts the agent creates, essentially building internal evaluation capabilities. ## Production Scale and User Behavior Since launching in September 2024, Replit Agent has processed "hundreds of thousands of runs in production." The team was surprised by several user behavior patterns: - Much higher than expected usage on mobile devices, with users building apps in two minutes on their phones - Extreme variance in prompt quality, from very concise and ambiguous prompts to users copy-pasting entire markdown files of instructions - Users spending hours on single projects with successful outcomes The focus on zero-to-one software creation—helping people with ideas quickly see prototypes—has proven effective. The team deliberately scoped down for launch, noting that working on pre-existing projects is "not very good" yet despite having the underlying technology. This narrow initial scope allowed them to iterate based on real user feedback rather than launching something generic that would receive more criticism. ## Key Lessons and Recommendations The talk concluded with several insights for others building similar systems: - **Bias towards action**: Don't fall into the paradox of choice with too many architectural options. Start building because models will change, requirements will evolve, and switching costs (both models and cognitive architectures) are not prohibitive. - **Scope isolation in multi-agent systems**: Having three to four agents with clear responsibilities keeps the system debuggable. More complexity risks losing control of system behavior. - **Observability from day one**: Essential for production agent systems. - **User-centered design over full autonomy**: Keeping users engaged and in the loop helps address the fundamental reliability challenges of agentic systems. The team noted that they're looking forward to future model improvements that might enable techniques like self-consistency (generating multiple LLM calls to check agreement) or Monte Carlo Tree Search-style approaches with parallel agent sessions—techniques that are currently cost-prohibitive but may become viable as model costs decrease.