## Overview This case study captures Anthropic's comprehensive approach to building production-ready AI agents, presented by Barry at an AI Engineer Summit. The presentation builds on a blog post written approximately two months prior titled "Building Effective Agents" and focuses on practical learnings from deploying agentic systems in production environments. Rather than promoting agents as universal solutions, Anthropic advocates for a disciplined, principled approach to determining when agents are appropriate and how to build them effectively. The presentation reflects real-world experience from building agents both internally at Anthropic and for customers, with particular emphasis on coding agents as a successful use case. ## Evolution of Agentic Systems The presentation situates agents within an evolutionary framework of LLM applications in production. The journey begins with simple single-call features like summarization, classification, and extraction that felt revolutionary two to three years ago but have since become baseline expectations. As practitioners gained sophistication, they moved to orchestrating multiple model calls in predefined control flows known as workflows. These workflows represent an explicit tradeoff: accepting higher costs and latency in exchange for improved performance. Anthropic positions workflows as the beginning of agentic systems rather than a separate category. The current phase involves domain-specific agents that can decide their own trajectory and operate with significant autonomy based on environment feedback. Unlike workflows where the control flow is predetermined, agents make dynamic decisions about their next steps. Looking forward, Anthropic anticipates either increasingly general-purpose single agents or collaborative multi-agent systems involving delegation and specialization. The overarching trend shows that as systems gain more agency, they become more useful and capable, but this comes with corresponding increases in cost, latency, and the potential consequences of errors. ## First Principle: Selective Agent Deployment Anthropic's first core principle challenges the notion that agents should be a default upgrade for every use case. Instead, agents should be viewed as a mechanism for scaling complex and valuable tasks, not as drop-in replacements for simpler solutions. The presentation introduces a structured checklist for evaluating whether a use case warrants an agent-based approach. **Task Complexity Assessment**: Agents thrive in ambiguous problem spaces where the solution path isn't predetermined. If you can easily map out the entire decision tree for a task, building that tree explicitly and optimizing each node will be more cost-effective and provide greater control. The explicit workflow approach allows for targeted optimization at each decision point rather than relying on the agent to discover the optimal path through exploration. **Value Justification**: The exploration inherent in agent behavior is token-intensive, so the task must justify the computational cost. For high-volume, low-budget scenarios like customer support systems with a budget of around 10 cents per interaction, you can only afford 30,000 to 50,000 tokens. In such cases, workflows designed to handle the most common scenarios will capture the majority of value more economically. Conversely, if the reaction to cost considerations is indifference to token consumption as long as the task gets done, agents become more viable. This honest admission about cost thresholds reflects real production constraints. **Capability Validation**: Before committing to an agent approach, it's essential to validate that there are no critical bottlenecks in the agent's required capabilities. For a coding agent, this means verifying that the model can write good code, debug effectively, and recover from its own errors. While bottlenecks aren't necessarily fatal, they multiply cost and latency as the agent struggles with weak areas. When bottlenecks are identified, the recommended approach is to reduce scope and simplify the task rather than pushing forward with a compromised solution. **Error Cost and Discovery**: High-stakes errors that are difficult to detect make it challenging to trust agents with autonomous action. While mitigation strategies exist, such as limiting agents to read-only access or implementing human-in-the-loop validation, these measures also limit the agent's ability to scale and operate independently. The fundamental tradeoff between autonomy and safety must be explicitly considered for each use case. The presentation uses coding agents as an exemplar of an ideal agent use case. Going from a design document to a pull request is inherently ambiguous and complex. Good code has clear value that justifies the investment. Cloud (Anthropic's LLM) has demonstrated strength across many parts of the coding workflow. Most importantly, coding has an exceptional property for production AI systems: outputs are easily verifiable through unit tests and continuous integration pipelines. This built-in verification mechanism addresses the error discovery problem that plagues many other agent use cases. ## Second Principle: Radical Simplicity Once an appropriate use case is identified, Anthropic's second principle advocates for keeping the implementation as simple as possible. The core architecture consists of three components: an environment (the system in which the agent operates), a set of tools (providing an interface for action and feedback), and a system prompt (defining goals, constraints, and ideal behavior). The model is then called in a loop, and that's the entire agent architecture in its simplest form. This emphasis on simplicity stems from hard-won experience that complexity kills iteration speed. The highest return on investment comes from iterating on these three basic components rather than adding architectural sophistication upfront. Optimizations should come later, after the fundamental behaviors are established. The presentation showcases three different agent use cases built for Anthropic or their customers that appear very different in terms of product surface, scope, and capabilities, but share nearly identical backbones and even similar code. Since the environment is largely determined by the use case itself, the primary design decisions reduce to selecting the tool set and crafting the prompt. **Optimization After Validation**: Once the basic components are working, various optimizations become relevant depending on the use case. For coding and computer use agents, caching the trajectory can reduce costs. For search agents with many tool calls, parallelizing those calls can reduce latency. Across nearly all agent applications, presenting the agent's progress in a way that builds user trust is critical. But these optimizations should follow, not precede, establishing the core behaviors. The presentation references the Model Context Protocol as an important tool framework worth exploring, indicating Anthropic's engagement with standardized tool interfaces for agents. ## Third Principle: Think Like Your Agent The third core principle addresses a common failure mode: developers building agents from their own perspective and becoming confused when agents make decisions that seem counterintuitive to humans. Anthropic recommends literally putting yourself in the agent's context window to understand what information is actually available at decision time. Despite exhibiting sophisticated behavior, agents at each step are simply running inference on a limited set of context, typically 10,000 to 20,000 tokens. Everything the model knows about the current state of the world must be explained within that context window. By limiting your own perspective to what's in that context, you can assess whether it's sufficient and coherent, bridging the gap between human understanding and agent perception. **The Computer Use Thought Experiment**: The presentation walks through an exercise of imagining being a computer use agent. You receive a static screenshot and a poorly written description. You can think and reason all you want, but only tool calls actually affect the environment. When you attempt a click, it's equivalent to closing your eyes for three to five seconds and using the computer in the dark during inference and tool execution. Then you open your eyes to another screenshot, uncertain whether your action succeeded or catastrophically failed. This cycle repeats continuously. Going through a full task from the agent's perspective is described as fascinating and only mildly uncomfortable, but highly revelatory. It quickly becomes clear what context the agent actually needs. For computer use, knowing the screen resolution is crucial for accurate clicking. Recommended actions and limitations help establish guardrails that reduce unnecessary exploration. This exercise should be customized for each agent use case to determine what contextual information is truly necessary. **Using LLMs to Understand LLMs**: Since these systems speak natural language, you can leverage them to improve themselves. You can provide your system prompt to the LLM and ask whether any instructions are ambiguous or difficult to follow. You can share tool descriptions and ask whether the agent understands how to use the tool and whether it would benefit from more or fewer parameters. A particularly useful technique involves providing the entire agent trajectory and asking why a particular decision was made and what could help improve decision-making. This shouldn't replace your own understanding of the context, but it provides valuable insight into how the agent perceives the world. ## Production Considerations and Future Directions The presentation concludes with forward-looking perspectives on open questions in production agent deployment. Three areas occupy significant attention: **Budget Awareness**: Unlike workflows where cost and latency are relatively predictable and controllable, agents lack clear mechanisms for enforcing budgets. Solving this problem would enable many more production use cases by providing the control necessary for confident deployment. The open question is determining the best way to define and enforce budgets, whether in terms of time, money, tokens, or other relevant metrics. **Self-Evolving Tools**: Building on the practice of using models to iterate on tool descriptions, there's potential for a meta-tool where agents design and improve their own tool ergonomics. This would make agents more general-purpose by allowing them to adapt tools to specific use case requirements. The concept represents a form of self-improvement that could accelerate agent adaptation to new domains. **Multi-Agent Collaboration**: The presentation expresses conviction that multi-agent collaborations will become significantly more common in production by the end of 2025. Multi-agent architectures offer natural parallelization, clean separation of concerns, and context window protection where sub-agents handle specialized tasks without polluting the main agent's context. However, a major open question concerns how agents should communicate with each other. Current systems are built around rigid, mostly synchronous user-assistant turns. Expanding to asynchronous communication and enabling different roles that allow agents to recognize and interact with each other represents a significant design challenge. ## Critical Assessment While the presentation offers valuable practical guidance, several important considerations deserve attention. The emphasis on simplicity is admirable but may risk underestimating the complexity that emerges at scale. Production systems often accumulate requirements that push against simplicity, and the transition from simple prototypes to robust production systems can involve significant architectural changes that aren't fully addressed. The checklist approach to agent suitability is helpful but may oversimplify the tradeoffs. The interaction between the four factors (complexity, value, capability, and error cost) can create edge cases where the decision isn't clear-cut. Organizations may find that factors like organizational readiness, monitoring capabilities, and incident response procedures also matter significantly. The discussion of cost optimization focuses primarily on token budgets but doesn't deeply engage with the full lifecycle costs of agent systems, including development time, maintenance burden, monitoring infrastructure, and the cost of building trust through gradual rollout. The comparison between agent costs and workflow costs could be more nuanced, as workflows also require significant engineering investment. The perspective-taking principle is valuable but may be challenging to operationalize consistently across a team. Tooling to make agent context windows more inspectable and debuggable would help institutionalize this practice rather than relying on manual thought experiments. The future directions identify important open problems but don't fully grapple with the governance and safety challenges that multi-agent systems introduce. As agents gain more autonomy and collaborate with each other, questions of accountability, auditability, and failure mode analysis become more complex. ## Strengths and Production Insights Despite these caveats, the presentation offers several notable strengths for LLMOps practitioners. The evolutionary framing from simple features through workflows to agents provides a valuable mental model for thinking about when to increase system complexity. The explicit acknowledgment that workflows remain highly valuable even as agents emerge prevents premature adoption of more complex architectures. The emphasis on verifiability as a key property for agent use cases is particularly important. The reason coding agents work well isn't just that models are good at coding, but that code has built-in verification mechanisms. This insight should guide use case selection more broadly toward domains with clear feedback mechanisms. The shared backbone across diverse agent applications suggests that standardization and reusable infrastructure are more achievable than they might appear. Organizations can invest in robust agent frameworks that apply across multiple use cases rather than building bespoke solutions for each application. The practice of using LLMs to evaluate and improve LLM systems represents a pragmatic approach to the challenge of understanding model behavior. While not a complete solution, it provides actionable insights that complement traditional debugging and evaluation approaches. Overall, this case study reflects Anthropic's practical experience deploying agents in production while maintaining realistic expectations about their capabilities and limitations. The emphasis on discipline in use case selection, simplicity in design, and empathy for the agent's perspective provides actionable guidance for organizations navigating the transition from experimental agent prototypes to production systems.