LangChain built LangSmith Engine, an autonomous agent designed to solve the persistent challenges their engineering team faced when maintaining production agents. The core problem was that manually monitoring agent traces was error-prone and time-consuming, with many small bugs getting deprioritized and micro-regressions occurring due to the difficulty of creating comprehensive evaluations. LangSmith Engine automatically identifies errors from production traces, generates fixes including code changes and prompt adjustments, creates custom online evaluators to catch recurring issues, and builds offline evaluation datasets from production failures. Early customers including Clay, Vanta, and Campfire reported that Engine fundamentally changed how they engineer their agents, making the agent improvement loop substantially more autonomous and efficient.
LangChain developed LangSmith Engine as an internal solution to challenges they encountered while building and maintaining production agents, particularly a go-to-market agent that performs account research and customer outreach. This case study provides valuable insights into meta-level LLMOps: building an agent that improves other agents. The team encountered familiar problems that many organizations face when deploying LLMs in production—despite careful monitoring, issues would slip through, small fixes would accumulate and get deprioritized, and creating evaluations to prevent regressions proved extremely tedious. The result was a cycle of micro-regressions that felt like “whack-a-mole” where fixing one problem would cause another to emerge.
LangSmith Engine represents their solution: an autonomous agent that identifies issues from production traces, generates fixes, and creates both online and offline evaluations to prevent future regressions. The case study is particularly valuable because LangChain documents their iteration process, architectural decisions, evaluation strategies, and key learnings from deploying this system both internally and with early customers.
The team identified three interconnected challenges in their agent operations. First, manual trace review was extremely error-prone. Even with meticulous monitoring of their production agents, the go-to-market team would inevitably report issues that slipped through observation. This highlights a fundamental challenge in LLMOps: the volume and complexity of agent traces makes comprehensive manual review practically impossible.
Second, while individual fixes were often straightforward, the sheer number of small issues created a prioritization problem. Each minor fix seemed low-priority in isolation, but collectively they significantly impacted agent performance. This is a common pattern in production LLM systems where the long tail of edge cases creates substantial operational overhead.
Third, the evaluation creation bottleneck created a vicious cycle. Without good evaluations, teams couldn’t confidently deploy fixes without risking regressions. But creating comprehensive evaluations for each fix was so time-consuming that it often didn’t happen, leading to the micro-regressions they were trying to avoid. This evaluation gap is perhaps the most critical challenge highlighted in the case study, as it prevents teams from scaling their agent improvement efforts.
LangSmith Engine operates as a multi-stage pipeline with sophisticated orchestration. The system is triggered either on a schedule or ad hoc, hitting the LangSmith backend which feeds into multi-tenant orchestration with a distributed task queue. Notably, all customer agents being improved by Engine run on a single LangSmith deployment, demonstrating the multi-tenant architecture required for this type of service.
The architecture deliberately separates issue identification from fix generation. This design decision emerged from early experience showing that the massive context required to investigate traces made it more effective to split these concerns. Deep agents run within LangSmith Sandboxes—one sandbox per run—pulling in both traces and the source code of the agents being improved. This sandboxed approach provides isolation and safety when analyzing potentially problematic code.
The system surfaces a prioritized inbox of issues in the UI, with each issue showing a description and timeline indicating whether it’s a new problem or long-standing. Users can drill into specific traces directly from the interface to verify whether an issue is legitimate before accepting proposed fixes. This human-in-the-loop design acknowledges that full automation isn’t always appropriate, especially for critical production systems.
A key architectural principle is that traces are far and away the most valuable input to Engine. While the agent’s source code is useful, it doesn’t effectively predict how the agent will perform in production due to the inherent non-determinism in agent systems. The team describes traces as “the window to the agent’s soul” for understanding real-world performance.
However, Engine doesn’t ingest entire traces verbatim. Instead, it receives condensed and summarized versions, with the ability to grab and investigate specific traces more closely when patterns seem interesting or potentially erroneous. This approach balances comprehensive coverage with computational efficiency, a critical consideration for production LLMOps systems.
The trace-first philosophy reflects a mature understanding of LLMOps: production behavior diverges from development expectations in complex ways, and observability infrastructure that captures actual execution is essential for continuous improvement.
Engine generates several categories of fixes. Prompt adjustments represent one common fix type, addressing issues through careful prompt engineering—which the team emphasizes is “still very much alive” despite claims to the contrary. The system also adjusts agent configuration files, including skills files and markdown files that define agent behavior.
More substantially, Engine can modify the actual code structure of agents. This code generation capability raises the stakes for accuracy, as incorrect code changes could break production systems. The case study doesn’t detail all the safeguards around code changes, but the PR-based workflow provides a review gate before deployment.
Beyond direct fixes, Engine contributes back to the evaluation suite in two ways. First, it creates custom online evaluators tailored to specific issues, ensuring that recurring problems get caught automatically in production. Second, it generates offline evaluation examples by taking production inputs that generated bad outputs and creating ground truth reference outputs. This evaluation dataset creation is described as particularly valuable, as manually creating ground truth for production failures is extremely tedious.
The team candidly describes Engine’s evolution from a “wind-up toy” to a sophisticated system. The first version was admittedly hacky: it would take LangSmith traces, send them to a coding agent, and spin up PRs via GitHub Actions running continuously. Despite the rough implementation, this early version surfaced real issues, including a tool call failing silently in one of their agents where Engine added appropriate error handling.
An even more interesting early success involved Engine detecting that users were requesting PR review functionality from their asynchronous coding agent called OpenSWE, and the agent was responding that it lacked this capability. Engine not only identified the unsatisfactory user experience but created a PR to extend the agent’s functionality. This demonstrated that Engine could do more than fix bugs—it could proactively improve agent capabilities based on observed user needs.
However, the first version had a critical flaw: it was too good at finding problems that weren’t real. The team references a Soviet-era quote—“Show me the man and I’ll show you the crime”—to describe how Engine would find issues in traces where none existed. This highlights a key challenge in automated issue detection: false positives can overwhelm teams and destroy trust in the system.
This experience led to the insight that identifying real and meaningful issues and clustering them into actionable buckets was the most important part of the entire process. The quality of fix generation and evaluation creation depended fundamentally on accurate issue identification. This learning shaped their architectural focus on separating and optimizing the issue identification stage.
A crucial learning emerged from their initial customer deployments: what constitutes a high-priority issue varies dramatically across teams and contexts. One customer praised Engine for catching hallucinations and data leaks but asked them to stop flagging bloated context, as their agent worked fine despite this. Another customer valued cost and latency optimization but didn’t care about minor hallucinations.
This variability demonstrates that production LLM systems exist in diverse operational contexts with different priorities, constraints, and risk tolerances. A one-size-fits-all approach to issue detection and prioritization doesn’t work.
Engine addresses this through customizable preferences. Users can change issue priorities, ignore specific issues with reasoning, or provide natural language feedback. All of this feeds into an “agent overview” that functions as a memory file, helping Engine understand what each customer cares about and how their specific agent works. This adaptive approach represents sophisticated LLMOps, where the improvement system itself learns from user feedback to better align with organizational priorities.
The evaluation approach for Engine itself provides valuable insights into evaluating complex agent systems. The team sources evaluations from three main channels. First, dogfooding—using Engine on their own internal agents like OpenSWE and their GTM agent—provides real user data. Since they are the users, they can quickly assess whether Engine is performing well and turn problematic traces into evaluations. This bootstrap approach is highly effective when you can be your own user.
Second, synthetic data generation using LLMs to create evaluation examples. However, the team emphasizes that LLMs are “really smart, but they’re actually very bad at this task” of generating good evaluations. Synthetic evals require extensive rubrics, guidance, and human review to ensure quality. This candid assessment is valuable, as synthetic evaluation generation is often presented as a silver bullet.
Third, community evaluations provide gut-checks, but the team found that bespoke evaluations tailored to their specific agent work best for measuring Engine’s performance. This suggests that while community evals are useful for sanity checks, production systems require custom evaluation suites that reflect their unique requirements.
Beyond sourcing, the team emphasizes building a diverse and well-rounded evaluation suite. Evaluations should cover different trace characteristics—short versus long traces, different domains like finance versus medical. Scenario-based tasks measure specific capabilities like tool calling. End-to-end evaluations using representative traces measure the full loop, enabling both holistic assessment and sub-component optimization.
These evaluations inform practical decisions including model selection as new models continually release, context engineering to design optimal tool sets, and prompt engineering to encode customer preferences. The rigorous eval-driven approach demonstrates mature LLMOps practices.
Despite the rigorous evaluation infrastructure, the team acknowledges an important role for qualitative assessment or “vibes.” Early in Engine’s development, before comprehensive evaluations existed, dogfooding and user feedback were critical. The team articulates a key principle: trust user feedback. If a customer reports that Engine is malfunctioning and evaluations suggest it’s fine, it’s probably malfunctioning, and the evaluation suite needs updating to capture the issue.
This balanced perspective—rigorous quantitative evaluation combined with responsiveness to qualitative feedback—reflects practical LLMOps wisdom. Evaluations are essential but imperfect proxies for real-world performance, and user feedback provides critical signals that evals might miss.
The most ambitious direction teased in the case study is Engine improving itself. Since Engine is an agent that produces traces like any other agent, those traces can be fed back into a version of Engine to generate improvement suggestions. The team describes seeing “first sparks” of this self-improving loop, with Engine identifying issues in its own context management, model selection, and other aspects.
This meta-level capability represents a fascinating frontier in LLMOps: autonomous systems that not only improve other systems but continuously improve themselves. However, the case study doesn’t provide detailed results from this self-improvement approach, noting only that more will be shared soon.
The system leverages multiple LangChain products including their deep agents framework and newly announced LangSmith Sandboxes. The distributed architecture with multi-tenant orchestration, task queues, and sandbox isolation demonstrates the infrastructure complexity required for production agent systems.
All of Engine’s execution is traced via LangSmith, demonstrating the critical role of observability in understanding agent performance. This recursive use of their own observability platform for developing their observability-improvement platform is a strong form of dogfooding.
Engine was deployed with approximately fifteen early customers including Clay, Vanta, and Campfire. These customers reportedly provided feedback that Engine changed not just how they interact with LangSmith, but how they engineer their agents entirely, making the process substantially easier. However, the case study lacks specific quantitative metrics on customer impact—no data on issues detected, false positive rates, fix acceptance rates, or time savings.
This absence of concrete metrics is notable in a presentation heavily emphasizing evaluation and measurement. It’s unclear whether this reflects the early stage of customer deployments, the challenge of defining appropriate metrics, or a decision to focus on qualitative feedback for the announcement.
The case study represents LangChain demonstrating their own capabilities by building a sophisticated product using their platform, which serves both as customer validation and marketing. While the technical architecture and learnings appear sound, several considerations warrant attention.
The claim that Engine changes how customers engineer their agents entirely is significant but unsubstantiated with concrete metrics or detailed case studies. The effectiveness of automated issue detection, fix generation, and evaluation creation likely varies substantially across different types of agents, codebases, and operational contexts.
The false positive problem in early versions raises important questions about current accuracy. How often does Engine identify real versus spurious issues? What’s the cognitive overhead of reviewing suggested fixes? For teams already stretched thin, a system that generates many low-value suggestions could create more burden than benefit.
The approach of generating code changes to production agents is inherently risky. While the PR-based workflow provides a review gate, teams must carefully assess each change, which may limit the time savings. The case study doesn’t address strategies for ensuring generated code is safe, maintainable, and aligned with architectural standards.
The customization based on user preferences is elegant, but it also introduces complexity. Different teams flagging different priorities means Engine must maintain separate context for each customer, and the quality of adaptation depends on the volume and clarity of feedback provided. It’s unclear how much interaction is required to adequately tune Engine for a specific team.
The evaluation strategy, while comprehensive, highlights how much work remains in evaluating agents. Synthetic evaluations require extensive human review, community evals don’t directly transfer, and dogfooding only works if you’re your own user. Most organizations building agents won’t have LangChain’s evaluation infrastructure or expertise, which may limit their ability to verify Engine’s effectiveness for their use cases.
The self-improvement loop teased at the end is compelling but highly speculative. The “first sparks” mentioned suggest very early experimentation, and the challenges of creating truly self-improving systems without human oversight are substantial.
This case study provides several valuable LLMOps insights. The observation that manual trace review doesn’t scale reflects a fundamental challenge in production LLM operations. As agent complexity and deployment scale increase, automated tooling for trace analysis becomes essential.
The separation of issue identification from fix generation demonstrates architectural thinking about where to invest optimization effort. By identifying issue detection as the critical bottleneck, they could focus resources appropriately.
The emphasis on traces as the primary signal reflects mature production operations. Development-time testing and code review cannot predict the emergent behaviors of complex agent systems in production with real users and data. Comprehensive observability that captures actual execution is foundational.
The integration of fix generation with evaluation creation addresses a key LLMOps challenge: the evaluation gap prevents confident iteration. By automatically generating evaluations alongside fixes, Engine reduces the friction in the improvement loop.
The customization based on user preferences acknowledges that LLMOps priorities vary by context. Cost, latency, accuracy, safety, and other dimensions have different importance across applications, and tooling must accommodate this diversity.
The rigorous evaluation of the evaluation system itself—using dogfooding, synthetic data, and community evals—demonstrates the recursive nature of LLMOps infrastructure development. Tools for improving production agents must themselves be developed and operated with production-grade rigor.
Overall, this case study illustrates both the promise and complexity of autonomous agent operations. While LangSmith Engine represents sophisticated automation of critical LLMOps workflows, realizing that potential requires substantial infrastructure, careful evaluation, and ongoing refinement based on production experience. The honest discussion of early failures, architectural evolution, and the continuing importance of human judgment provides valuable context for organizations considering similar approaches to agent observability and improvement.
Cursor, a developer tool company, shares their journey of building what they call a "software factory" where AI agents handle increasingly autonomous software development tasks. The presentation outlines how they progressed through levels of autonomy from basic autocomplete to spawning hundreds of agents working asynchronously across their codebase. Their solution involves establishing guardrails through rules that emerge dynamically, creating verifiable systems with automated testing, and building skills and integrations that enable agents to work independently. Results include engineers managing fleets of agents rather than writing code directly, with some features being developed entirely by agents from feature flagging through testing to deployment, though significant work remains in observability, orchestration, and preventing agents from going off-track.
Notion, a knowledge work platform serving enterprise customers, spent multiple years (2022-2026) iterating through four to five complete rebuilds of their agent infrastructure before shipping Custom Agents to production. The core problem was enabling users to automate complex workflows across their workspaces while maintaining enterprise-grade reliability, security, and cost efficiency. Their solution involved building a sophisticated agent harness with progressive tool disclosure, SQL-like database abstractions, markdown-based interfaces optimized for LLM consumption, and a comprehensive evaluation framework. The result was a production system handling over 100 tools, serving majority-agent traffic for search, and enabling workflows like automated bug triaging, email processing, and meeting notes capture that fundamentally changed how their company and customers operate.
LangChain developed LangSmith Engine, an autonomous agent that continuously monitors production agent traces, identifies failures, clusters them into actionable issues, and proposes code fixes. The problem was the manual, time-intensive process of agent debugging where engineers had to manually review traces, identify patterns, write fixes, and run evaluations. Engine automates this cycle by running as an always-on background agent that ingests production traces, uses hierarchical agent architectures with specialized sub-agents to analyze millions of traces cost-effectively, surfaces prioritized issues in an inbox-style interface, and generates pull requests with proposed fixes along with evaluation dataset examples. Since launching publicly about a month ago after iterative development with design partners, Engine has become a core part of LangChain's agent development workflow, with the team even running Engine on Engine's own traces to find improvements in a meta-recursive improvement loop.
