DoorDash's development and deployment of AutoEval represents a sophisticated example of operationalizing LLMs in a production environment to solve a critical business challenge: evaluating search result quality at scale. This case study demonstrates several key aspects of LLMOps including prompt engineering, model fine-tuning, human-in-the-loop processes, and production deployment considerations. The core problem DoorDash faced was the inefficiency and lack of scalability in their traditional human-based search evaluation process. With their search system expanding to support multiple verticals (restaurants, retail, grocery, pharmacy), manual evaluation became a significant bottleneck. Human annotations were slow, inconsistent, and provided limited coverage, especially for tail queries where relevance issues often lurk. AutoEval's architecture showcases several important LLMOps practices: First, the system implements a sophisticated prompt engineering approach. Rather than using simple zero-shot prompts, DoorDash developed structured templates that incorporate domain-specific context and evaluation criteria. They use chain-of-thought reasoning to break down evaluation tasks into discrete steps, mimicking human evaluator decision processes. The prompts include rich metadata and embedded guidelines, demonstrating how careful prompt design can help LLMs perform complex domain-specific tasks. The system employs a hybrid approach to model deployment, combining both base and fine-tuned models. DoorDash fine-tunes models on expert-labeled data for key evaluation categories, with special attention to capturing not just the labels but also the reasoning behind them. This shows how domain adaptation can be achieved through targeted fine-tuning while maintaining model interpretability. A critical aspect of AutoEval's success is its human-in-the-loop framework. The system isn't designed to completely replace human judgment but rather to augment it. Expert raters focus on high-value tasks like guideline development and edge case resolution, while external raters audit the system's output. This creates a continuous feedback loop for system improvement, with experts analyzing flagged outputs to refine prompts and training data. The production deployment includes several noteworthy technical elements: * Query sampling from live traffic across multiple dimensions * Structured prompt construction for different evaluation tasks * LLM inference with both base and fine-tuned models * Custom metric aggregation (WPR) * Continuous monitoring and auditing The Whole-Page Relevance (WPR) metric represents an innovative adaptation of traditional relevance metrics for modern search interfaces. Unlike simple list-based metrics like NDCG, WPR considers the spatial layout and visual prominence of different content blocks, providing a more realistic measure of user experience. From an LLMOps perspective, DoorDash's future roadmap highlights important considerations for scaling and maintaining LLM-based systems: * They plan to implement an internal GenAI gateway to abstract away direct dependencies on specific LLM providers, enabling easier experimentation with different models * They're exploring in-house LLMs for better control and cost efficiency * They're working on enhancing prompt context with external knowledge sources The results demonstrate the potential of well-implemented LLMOps: * 98% reduction in evaluation turnaround time * 9x increase in evaluation capacity * Consistent matching or exceeding of human rater accuracy * More efficient use of expert human resources However, it's worth noting some potential limitations and challenges: * The system currently depends on commercial LLM APIs, which could present cost and control challenges at scale * The effectiveness of the system relies heavily on high-quality expert-labeled training data * The approach requires ongoing human oversight and auditing to maintain quality From an infrastructure perspective, the case study shows how LLMs can be integrated into existing workflows while maintaining quality control. The system architecture demonstrates careful consideration of scalability, reliability, and maintainability - all crucial aspects of production ML systems. DoorDash's implementation of documentation and quality control processes is notable. They maintain clear evaluation guidelines, document prompt engineering decisions, and have structured processes for model evaluation and refinement. This attention to MLOps best practices helps ensure the system remains maintainable and improvable over time. The success of AutoEval also highlights the importance of choosing the right problems for LLM automation. Rather than trying to completely automate search relevance decisions, DoorDash focused on augmenting human expertise and automating the more routine aspects of evaluation. This allowed them to achieve significant efficiency gains while maintaining or improving quality standards. Looking forward, DoorDash's plans to abstract their LLM infrastructure and potentially develop in-house models represent a mature approach to LLMOps. These steps would give them more control over their AI infrastructure while potentially reducing costs and improving performance for their specific use case. This case study demonstrates that successful LLMOps isn't just about deploying models - it's about building sustainable systems that combine machine learning capabilities with human expertise in a way that scales effectively while maintaining quality standards.