This case study examines best practices for AI evaluation in production systems, drawing on expertise from practitioners at ASU and Google. The discussion addresses the challenge of moving beyond informal "vibe checks" to establish rigorous evaluation frameworks that guide product development, ensure regulatory compliance, and build user trust. The solution emphasizes a team-based approach combining offline evaluation, online experimentation, manual data analysis, and statistical rigor including causal inference techniques. Results highlight that effective evaluation systems require alignment between product managers, engineers, and domain experts, with evaluation serving as both a compass for product iteration and a critical gate for release decisions, particularly in regulated industries like education.
This case study presents insights from Stella Liu, Head of Applied Science at Arizona State University (ASU), and Eddie Landsberg, Staff Data Scientist at Google and founder of Seemo Labs, on implementing rigorous AI evaluation practices for production LLM systems. The discussion, hosted by Hugo Bowne Anderson of Vanishing Gradients, explores the evolution of AI evaluation from informal testing to a mature discipline incorporating statistical rigor and causal inference. At ASU, the team builds AI platforms with a strong focus on product development rather than pure research, developing platform solutions specifically for AI evaluations in the regulated education sector.
The fundamental challenge addressed is that many organizations struggle to establish effective evaluation frameworks for their AI systems. Teams often rush to automation using vendor tools or generic metrics without understanding their data, defining clear product requirements, or ensuring alignment between stakeholders. This leads to optimizing proxy metrics that don’t correspond to actual business impact or user needs, particularly problematic as systems scale and regulatory requirements increase.
The practitioners emphasize that AI evaluation serves multiple critical functions in production environments. First, it creates essential feedback loops in the product development lifecycle, functioning as a compass to guide both initial development and ongoing iteration. Second, evaluations act as pass/fail gates for release decisions, determining whether products are ready for experimentation or full deployment. Third, particularly in regulated industries like education where ASU operates, evaluations verify compliance with regulatory requirements and safety standards.
The team strongly advocates that evaluation is fundamentally a team sport requiring tight collaboration between product managers, data scientists, UX designers, and subject matter experts (SMEs). While AI product managers tend to be quite technical, the scope of comprehensive evaluation exceeds any single person’s capability. Domain experts, even non-technical ones, bring crucial perspectives that purely technical teams might miss. The team composition becomes increasingly important as projects mature and scale.
The experts distinguish between offline evaluation and online experimentation, viewing them as complementary rather than competing approaches. Offline evaluation involves measuring products against expectations before they meet real users, essentially testing whether the system performs as the product team anticipates. This phase is critical for identifying obvious failure modes and building confidence before investing in more expensive online experiments.
Online evaluation through AB testing and randomized controlled trials provides ground truth about what real users actually need and value, which can differ significantly from product team assumptions. The practitioners frame this as policy evaluation, a problem that has been studied for hundreds of years in statistics and causal inference. As AI evaluation matures, it increasingly resembles traditional data science problems with established methodologies for measuring uncertainty, quantifying differences between variants, and determining statistical significance.
For early-stage startups not in heavily regulated industries, the team acknowledges that extensive evaluation may not be necessary in the first iteration. There’s limited risk and benefit in shipping quickly to learn. However, as companies build user bases and need to maintain trust with clients, evaluation becomes increasingly critical. The natural evolution involves allocating more resources and involving more people in evaluation projects as products mature.
The practitioners identify several critical mistakes teams make when implementing AI evaluation systems. The most significant failure mode is deploying vendor tools or open-source evaluation packages with built-in metrics without actually examining the data. Generic metrics typically lead to generic products that fail to differentiate or address specific user needs. Teams that skip manual data analysis before rushing to automation miss crucial insights about how their systems actually behave.
The lack of curiosity about data represents a fundamental problem. Teams sometimes want to use agents to analyze evaluation results rather than looking themselves, which raises questions about their understanding of what they’re building. Once people actually examine their data, they inevitably discover interesting patterns, unexpected behaviors, and failure modes that drive product improvements. This manual inspection phase is irreplaceable for building mental models of system behavior.
Another common failure is the absence of clear product requirements documents (PRDs) or specifications. Many teams move directly from prototyping to production without defining what the product should do and, critically, what it should not do. These foundational discussions about scope, capabilities, and constraints should happen early in development but frequently occur much later. Without clear expectations, teams cannot effectively evaluate whether systems meet their goals.
As evaluation systems mature and scale, they increasingly require statistical sophistication. The team emphasizes understanding non-deterministic behavior as fundamental to effective offline evaluation. Unlike traditional software testing with simple pass/fail outcomes, AI evaluation deals with distributions and confidence levels. Teams need mental models that account for randomness and approximation of the true distribution of user interactions.
Building good test sets requires understanding the distribution of real user interactions and mapping that to offline evaluation datasets. This sampling problem has deep statistical roots. Teams must also understand how much noise versus signal exists in their measurements and how many examples they need to confidently differentiate between system variants.
Eddie Landsberg’s work on causal judge evaluation (CJE) specifically addresses calibrating LLM judges to human experts or ground truth labels. Since high-quality human labels are expensive, teams can only afford limited quantities. LLM judges provide inexpensive labels at scale but aren’t necessarily aligned with human judgment or business objectives. The causal inference framework focuses on ensuring that automated metrics actually correspond to outcomes that matter, avoiding the trap of optimizing scores that don’t reflect real value.
The approach treats model iterations like randomized control trials, explicitly considering counterfactuals and parallel universes where different system configurations are deployed. This requires identifying confounders, understanding missing data problems, and making assumptions explicit. Bayesian methods receive particular emphasis because they force practitioners to state their assumptions clearly, unlike frequentist approaches where assumptions remain implicit and often unexamined.
The practitioners advocate for both qualitative and quantitative evaluation rather than forcing everything into numerical metrics. Most AI evaluation starts with qualitative assessment to identify which aspects are deterministic and automatable versus which remain inherently qualitative and require ongoing human review. While automation is desirable for scalability, some application aspects cannot and should not be quantified.
This balanced approach appears particularly important in domains like education where human judgment about learning effectiveness, appropriateness for age groups, and pedagogical quality cannot be fully captured by automated metrics. The team emphasizes that decisions ultimately need to be made, which often requires quantitative comparison, but the path to those decisions should incorporate rich qualitative understanding.
The discussion touches on using LLMs as judges for evaluation, emphasizing significant challenges with this approach. The convenience of LLM judges leads teams to deploy them in every situation without carefully considering how they map to human values and welfare. Generic benchmarks claiming to measure occupational skills or capabilities should receive scrutiny from real humans, not just LLM labels.
The emerging best practice involves grounding LLM judges in trusted human judgment through careful calibration. This requires collecting some subset of examples labeled by human experts and using statistical techniques to ensure the automated judge correlates with human assessment. The causal inference framework helps determine how many human labels are needed and how to use limited human feedback most efficiently.
At ASU, operating in the regulated education sector creates specific evaluation requirements. Stella Liu emphasizes the need for clearer industry standards and more specific regulatory guidance. Current regulations and procurement processes require evaluations but remain vague about what exactly constitutes adequate evaluation. Federal, state, and institutional guidance mentions evaluation expectations but lacks detailed specifications for compliance.
This ambiguity creates challenges for AI product teams trying to meet regulatory requirements. The team advocates for more concrete standards specifying what evaluations must include, how they should be conducted, and what thresholds must be met. As AI proliferates in regulated industries including healthcare, finance, and education, robust evaluation standards will become critical competitive advantages and gatekeepers for market access.
Before any automation or sophisticated statistical techniques, the team emphasizes thorough error analysis as foundational. This involves categorizing failure modes, identifying patterns in when and how systems fail, and understanding edge cases. Error analysis provides high-impact insights because it reveals obvious problems that clearly aren’t consistent with good products and that users will certainly notice.
The team notes that users sharing the same negative reactions to obvious failures makes error analysis a relatively safe starting point. More subtle aspects like tone, helpfulness, or creativity require more careful validation that product team intuitions match user preferences. This distinction guides where teams should invest in formal user research versus where they can rely on internal judgment.
Looking forward, the practitioners want to see AI evaluation systems that genuinely ground automated metrics in human values rather than using convenient but misaligned proxies. Benchmarks should reflect what humans actually care about, not just what’s easy to measure with LLM judges. This requires maintaining human involvement even as systems scale and continuing to validate that automated metrics predict real-world outcomes.
The field is moving toward greater maturity by adopting established practices from statistics, causal inference, and experimental design. AB testing methodologies, power analysis, confidence intervals, and careful consideration of confounders are all becoming standard practice in AI evaluation. The convergence with traditional data science represents positive evolution toward more rigorous and reliable evaluation systems.
The team also emphasizes that practitioners should think from first principles about what their users care about rather than following hype or feeling insecure about not using the latest tools. The most powerful evaluation tool remains human judgment and curiosity about how systems actually behave in practice. This foundation of caring about products and looking carefully at data should precede any sophisticated automation or tooling decisions.
From ASU’s platform development experience, several practical insights emerge. Teams should identify evaluation roles early, clarifying who owns which decisions to help reconcile disagreements. The concept of a benevolent dictator for evaluation can help, but teams should also consider whether disagreement between informed parties suggests averaging perspectives rather than choosing one.
Starting with clear product requirements that define both capabilities and constraints prevents evaluation scope creep and ensures alignment. Teams should understand what constitutes deterministic behavior versus what requires probabilistic assessment. They should build test sets that genuinely represent the distribution of real user interactions, not just convenient or easy-to-label examples.
The progression from manual to automated evaluation should be deliberate, automating what can be reliably automated while maintaining human judgment where it adds irreplaceable value. The feedback loop between looking at data, forming hypotheses about what matters, testing those hypotheses, and refining evaluation systems drives continuous improvement in both products and evaluation methodologies.
Throughout the discussion, the emphasis remains on evaluation as enabler of better products and user experiences rather than as bureaucratic overhead. When properly implemented, evaluation systems provide the compass for product development, the confidence for release decisions, and the evidence for regulatory compliance that together enable responsible deployment of AI systems at scale.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Rippling, an enterprise platform providing HR, payroll, IT, and finance solutions, has evolved its AI strategy from simple content summarization to building complex production agents that assist administrators and employees across their entire platform. Led by Anker, their head of AI, the company has developed agents that handle payroll troubleshooting, sales briefing automation, interview transcript summarization, and talent performance calibration. They've transitioned from deterministic workflow-based approaches to more flexible deep agent paradigms, leveraging LangChain and LangSmith for development and tracing. The company maintains a dual focus: embedding AI capabilities within their product for customers running businesses on their platform, and deploying AI internally to increase productivity across all teams. Early results show promise in handling complex, context-dependent queries that traditional rule-based systems couldn't address.
This case study presents lessons learned from deploying generative AI applications in production, with a specific focus on Flo Health's implementation of a women's health chatbot on the Databricks platform. The presentation addresses common failure points in GenAI projects including poor constraint definition, over-reliance on LLM autonomy, and insufficient engineering discipline. The solution emphasizes deterministic system architecture over autonomous agents, comprehensive observability and tracing, rigorous evaluation frameworks using LLM judges, and proper DevOps practices. Results demonstrate that successful production deployments require treating agentic AI as modular system architectures following established software engineering principles rather than monolithic applications, with particular emphasis on cost tracking, quality monitoring, and end-to-end deployment pipelines.
