Lyft's data science team developed a comprehensive evaluation framework for their customer support AI agent system over a two-year period, addressing the challenge of ensuring AI agents perform reliably before deployment to live users. The solution involves a multi-layered approach combining offline evaluations with synthetic conversation simulation, fine-tuned user models that realistically mimic actual customer behavior, actionable LLM-as-judge metrics tied to business outcomes, and rigorous statistical validation methods. The team achieved more reliable production deployments by replacing generic evaluation metrics with task-specific binary outcomes validated against human-labeled ground truth, while also establishing a continuous error analysis loop that feeds insights back into model improvements through context learning, harness tuning, and planned reinforcement learning approaches.
Lyft’s data science and engineering team, led by Nick and Ashe, developed a sophisticated end-to-end evaluation pipeline for their customer support AI agent system over approximately two years of development. The system represents a mature LLMOps implementation that treats AI agent deployment with the same rigor as traditional machine learning model deployment, emphasizing that live users should never serve as test data for AI systems. The team’s work draws inspiration from academic research, particularly Sierra AI’s TauBench paper for customer support agents and Microsoft’s UserAlen research on user simulation.
The evaluation system is structured around two distinct phases: development and production. During development, teams build agentic components including context management, retrieval pipelines, tool definitions, agentic graphs using LangGraph, and system prompts. The critical innovation is the introduction of a rigorous offline evaluation process that acts as a gate before production deployment, mirroring traditional ML model validation practices but adapted for the multi-turn, conversational nature of AI agents.
The offline evaluation pipeline prevents the anti-pattern of using live customers as test subjects, which the team emphasizes is poor practice regardless of the application domain. Once agents pass offline evaluation criteria and deploy to production, an online evaluation pipeline continuously monitors performance through tracing tools, online graders, and human-in-the-loop error analysis that feeds insights back to development teams for continuous improvement.
The core of the offline evaluation system involves simulating realistic multi-turn conversations between the AI agent and a synthetic user model. The team identifies three essential components: a representative synthetic dataset, an LLM playing the role of the user to generate complete multi-turn simulated conversations, and a grader such as an LLM-as-judge to evaluate interaction quality.
A critical learning emerged around synthetic data generation. The team warns against the common mistake of simply prompting an LLM to generate test queries, as this produces unrealistic data that doesn’t reflect production conditions. Instead, Lyft adopts a hybrid approach: sampling actual production conversations from their extensive historical customer support data and mutating different criteria to cover both golden paths and edge cases. This ensures the offline dataset closely resembles real-world traffic patterns and customer behavior.
One of the most significant technical challenges the team encountered was that frontier LLMs, when role-playing as customers, behaved too cooperatively and patiently. These models are trained as helpful assistants and consequently produced verbose, polite explanations of issues. In initial offline evaluations, this unrealistic behavior led to artificially inflated performance metrics with pass rates around 90 percent.
In reality, Lyft customers contacting support are typically impatient, frustrated, and use terse language rather than detailed explanations. To address this gap, the team fine-tuned an LLM specifically on real Lyft user verbatim to make the simulation model speak authentically like actual customers. This fine-tuning caused evaluation scores to decrease, which initially might seem counterproductive, but actually represents a major improvement in evaluation validity. Easy evaluations that don’t reflect production conditions provide false confidence and limited actionable insights.
Additionally, the team defined specific user personas such as bypass users who immediately demand human escalation without giving the AI a chance, refund seekers, and AI skeptics. These personas, inspired by Microsoft’s UserAlen research, ground the simulation in realistic behavioral patterns and help the system test agent robustness against difficult customer interactions.
The team’s experience with LLM-as-judge systems reveals important lessons about what makes evaluation actionable. Initially, they relied on pre-built metrics from frameworks like DeepEval, measuring dimensions such as tool usage appropriateness, response helpfulness, conversation naturalness, and completeness. While these metrics were technically relevant, they proved non-actionable. A score of 0.5 on response helpfulness provides no clear guidance on what needs to be fixed or improved.
The breakthrough came from reframing evaluation around task success or failure with binary outcomes. Binary classifications are significantly easier to calibrate and train LLM judges to score consistently. By collaborating closely with domain experts, the team developed metrics aligned with specific business goals and functional requirements. This partnership between domain experts and data scientists produces metrics that are both actionable and meaningful when failures occur, enabling systematic error pattern analysis.
One concrete example is their education rubric metric, which defines clear success and failure criteria. The agent fails if it tries too many times to educate the user when escalation would be appropriate, or if it escalates too soon without attempting user education. Conversely, it passes when following expected behavior patterns. This binary, task-oriented approach provides clear signals about what went wrong and how to fix it.
A critical innovation in Lyft’s approach is treating LLM judges themselves as binary classifiers that require validation. The team applies traditional machine learning evaluation techniques to their judges. They hand-label approximately 100 examples with pass/fail labels based on ground truth, then split this data into train, dev, and validation sets. The split percentages differ from traditional ML since they’re not actually training model weights but rather informing the judge’s prompt.
The training set provides few-shot examples included in the judge’s prompt. The dev set is used for iterative prompt refinement and judge improvement. The validation set ensures the judge hasn’t overfit to dev examples. By calculating precision and recall scores for each judge against human-labeled ground truth, the team obtains objective measures of judge quality and consistency. This statistical rigor is especially important when evaluation scores gate shipping decisions or when reporting to company leadership.
The team recognizes that evaluation criteria themselves evolve as they observe more data and grade more outputs. They emphasize that evaluation criteria cannot be defined upfront in isolation and then applied mechanically. Instead, criteria should co-develop with the model as teams test evaluators and calculate precision and recall. This iterative refinement process acknowledges that understanding of quality evolves with exposure to production data and edge cases. Evaluations should not be decoupled from model observations but rather developed alongside the model.
To make reported numbers more meaningful, the team advocates adding statistical rigor through confidence intervals, proper calibration, and appropriate sampling. They provide a concrete example: if one evaluator scores 84 percent and another scores 88 percent based on only 50 examples, the four percentage point gain is too small to be meaningful with such limited data. Demonstrating that a gain is real requires either larger effect sizes or more samples. The team reserves expensive statistical rigor for moments when numbers actually gate decisions or inform leadership, but emphasizes that every score needs an interval to be interpretable.
Comprehensive tracing forms the foundation for all evaluation activities. The team uses tools like LangSmith and LangFuse to log and view traces. Each trace captures the full graph execution including which nodes ran, what the LLM saw, which tools were called, token usage, and latency for every call. Traces can be enriched with metadata to provide additional insights beyond raw execution data.
The team also implements annotation queues that provide domain experts with an easy-to-understand UI for labeling and providing feedback on evaluator performance. Rather than forcing experts to parse raw JSON traces, annotation queues enable efficient labeling that becomes the ground truth for calculating judge precision and recall and for adding high-quality examples to offline evaluation datasets.
Rather than treating evaluation as a one-off audit, Lyft runs a continuous error analysis loop with regular cadence, typically weekly or bi-weekly. The process involves deep diving into raw traces, pinpointing specific failure modes, eliminating noisy metrics to focus only on those that change decisions, forming fresh premises for re-evaluation, and repeating the cycle. This continuous improvement mindset ensures evaluation criteria remain aligned with evolving product requirements and customer behaviors.
The team identifies several evaluation anti-patterns to avoid, with the most critical being ignoring the data. Without examining actual traces and user interactions, teams cannot create meaningful criteria or labels. Without labels, they cannot evaluate judges. Without evaluated judges, they cannot know whether agentic pipelines work as expected. Data examination acts as the foundation for meaningful evaluation throughout the entire system.
The ultimate purpose of evaluation is feeding insights back into model and system improvements. The team frames this as three types of learning: model learning, context learning, and harness learning. Model learning involves post-training updates to underlying model weights through fine-tuning or training custom LLMs. Context learning improves the information the agent sees, including documents, stored user memories, and tool outputs. Harness learning updates system prompts, tool schemas, control flow, routing logic, and retry mechanisms.
Error analysis directly informs improvements to agent prompts, knowledge base updates, and context management strategy refinement. Identifying failure modes provides actionable guidance for tuning both agent context and harness components. Looking forward, the team is exploring post-training approaches more deeply, leveraging the extensive real user signals collected over years of operation to fine-tune specialized models for different customer support tasks and to frame reward modeling problems that enable reinforcement learning.
To operationalize their evaluation philosophy, the team is building a systematic evaluation harness that runs offline evaluations in a standardized, repeatable manner. Currently, their evaluation scripts are scattered across notebooks and analysis repositories, making it difficult to maintain consistency and enable broad contribution.
The evaluation harness is config-driven, with evaluation suites stored as YAML files easily editable by contributors beyond just engineers, including analysts and data scientists. Key design principles include parallelism and high throughput to handle thousands or tens of thousands of examples in reasonable time, and well-defined primitives including tasks, datasets, personas, LLM adapters, and evaluators.
The harness enables a define-once, run-anywhere approach where evaluation configs can be executed at multiple touchpoints throughout development. Developers can run evaluations locally while tuning prompts to get immediate feedback on performance compared to previous versions. Evaluations can run at pre-commit hooks to prevent performance regressions before changes are pushed to agent services. The team is also exploring integration into CI/CD pipelines to build regression test suites and acceptance test suites that automatically validate changes.
The evaluation system serves as a critical launch gate between development and production. The team establishes specific criteria that offline evaluations must meet before an agent version can deploy to live users. This gating mechanism ensures confidence that agents have sufficient performance before customer exposure. The approach mirrors traditional ML deployment practices where models must pass offline validation metrics before serving production traffic.
In production, online evaluation pipelines continuously monitor deployed agents through real-time grading of agent performance and human-in-the-loop review processes. The combination of automated online grading and human error analysis creates a feedback loop that surfaces regression issues quickly and identifies clear owners responsible for taking action based on evaluator insights.
Several overarching principles emerge from Lyft’s experience. First, evaluations only matter when they meaningfully gate something. Scores that float without influencing decisions provide no value. Second, LLM judges must be validated with the same rigor as any other ML model, including precision and recall calculations against ground truth. Third, evaluation criteria must be actionable and tied to specific business outcomes rather than generic quality dimensions. Fourth, synthetic data for offline evaluation must realistically represent production conditions, even if this means lower scores. Fifth, evaluation is a continuous process requiring regular error analysis loops rather than one-time audits.
The team emphasizes that building meaningful evaluation systems requires significant time and effort. Teams cannot rely exclusively on LLMs for evaluation without human validation and statistical rigor. The data examination process, while time-consuming, is non-negotiable because it forms the foundation for everything else in the evaluation pipeline.
Beyond the evaluation harness, Lyft is investing in post-training capabilities to leverage the extensive real user feedback and interaction data they’ve collected. This includes fine-tuning specialized models for specific customer support tasks and developing reward models to enable reinforcement learning from human feedback. The combination of sophisticated evaluation infrastructure and advanced training techniques positions the team to continuously improve agent performance based on real-world deployment experience.
This comprehensive case study examines how multiple enterprises (Autodesk, KPMG, Canva, and Lightspeed) are deploying AI agents in production to transform their go-to-market operations. The companies faced challenges around scaling AI from proof-of-concept to production, managing agent quality and accuracy, and driving adoption across diverse teams. Using the Relevance AI platform, these organizations built multi-agent systems for use cases including personalized marketing automation, customer outreach, account research, data enrichment, and sales enablement. Results include significant time savings (tasks taking hours reduced to minutes), improved pipeline generation, increased engagement rates, faster customer onboarding, and the successful scaling of AI agents across multiple departments while maintaining data security and compliance standards.
Notion, a knowledge work platform serving enterprise customers, spent multiple years (2022-2026) iterating through four to five complete rebuilds of their agent infrastructure before shipping Custom Agents to production. The core problem was enabling users to automate complex workflows across their workspaces while maintaining enterprise-grade reliability, security, and cost efficiency. Their solution involved building a sophisticated agent harness with progressive tool disclosure, SQL-like database abstractions, markdown-based interfaces optimized for LLM consumption, and a comprehensive evaluation framework. The result was a production system handling over 100 tools, serving majority-agent traffic for search, and enabling workflows like automated bug triaging, email processing, and meeting notes capture that fundamentally changed how their company and customers operate.
AI practitioners Aishwarya Raanti and Kiti Bottom, who have collectively supported over 50 AI product deployments across major tech companies and enterprises, present their framework for successfully building AI products in production. They identify that building AI products differs fundamentally from traditional software due to non-determinism on both input and output sides, and the agency-control tradeoff inherent in autonomous systems. Their solution involves a phased approach called Continuous Calibration Continuous Development (CCCD), which recommends starting with high human control and low AI agency, then gradually increasing autonomy as trust is built through behavior calibration. This iterative methodology, combined with a balanced approach to evaluation metrics and production monitoring, has helped companies avoid common pitfalls like premature full automation, inadequate reliability, and user trust erosion.
