## Overview Wayfair, a major e-commerce retailer specializing in home goods, developed Agent Co-pilot, a generative AI system designed to assist their digital sales agents during live customer chat interactions. Unlike simple rule-based chatbots, this system operates as an AI copilot that provides real-time response recommendations to human agents, who then have the final decision on whether to use, modify, or discard the suggestions. This human-in-the-loop approach represents a pragmatic deployment strategy that balances AI capabilities with human oversight and quality control. The core business problem being addressed is improving customer service efficiency while maintaining quality. When customers need personalized help—whether asking product questions or seeking assistance in finding the right items—agents must quickly access relevant product information, company policies, and craft appropriate responses. Agent Co-pilot aims to reduce the cognitive load on agents by surfacing relevant information and generating draft responses in real-time. ## Technical Architecture and Prompt Engineering The system's architecture centers on a carefully constructed prompt that feeds into a Large Language Model. The prompt engineering approach is multi-faceted, incorporating several key components that work together to produce contextually appropriate responses. The prompt structure includes a task description that explicitly defines what the LLM should accomplish—such as providing product information, clarifying return policies, or suggesting alternative products. This is complemented by guidelines that outline internal processes agents must follow, ensuring the AI-generated responses align with established service standards. Company policies related to shipping, returns, and assembly services are also embedded in the prompt to ensure responses reflect current business rules. Product information is dynamically included when customers inquire about specific items, enabling the LLM to answer product-related questions accurately. Crucially, the system maintains and incorporates conversation history, moving beyond single-turn interactions to provide contextually relevant suggestions that account for the full dialogue context. This multi-turn capability is essential for handling realistic customer service scenarios where context builds over time. The response generation process follows standard autoregressive LLM behavior—the model predicts the most likely next token based on patterns learned during training, iteratively building a complete response. What's notable here is the emphasis on the prompt as the primary mechanism for controlling model behavior, rather than relying on fine-tuned models (though fine-tuning is mentioned as a future direction). ## Quality Monitoring and Evaluation One of the more sophisticated aspects of this deployment is the comprehensive quality monitoring framework. The team employs both quantitative and qualitative evaluation methods, which is essential for production LLM systems where automated metrics alone may not capture all aspects of response quality. The quality metrics framework includes prompt instruction adherence, which tracks how closely Co-pilot's responses follow specific instructions in the prompt. This could include constraints on response length, required greetings, or closing templates. By monitoring rule breaks over time, the team can identify failure modes and assess system stability—a practical approach to understanding where the LLM struggles to follow explicit instructions. Factuality evaluation addresses the critical issue of hallucinations, verifying that product information, policy details, and other data in responses are accurate. This is particularly important in e-commerce where incorrect product specifications or policy information could lead to customer dissatisfaction or operational issues. The edit reason tracking provides valuable feedback on why agents modify Co-pilot suggestions before sending them to customers. Categories include stylistic changes, missing product information, policy adherence issues, and data correctness problems. This human feedback loop is essential for understanding real-world performance gaps that automated metrics might miss. Message purpose analysis categorizes responses by intent (answering questions, providing product info, suggesting alternatives, etc.) and compares the distribution of Co-pilot's purposes with actual agent behavior. This helps identify where the AI's behavior diverges from human patterns and may need adjustment. An interesting addition is the use of a secondary "QA LLM" to assess Co-pilot response quality. This LLM-as-judge approach has become increasingly common in production systems, providing scalable automated evaluation, though it comes with its own limitations around evaluator bias and the need to validate that the QA LLM's assessments correlate with human judgments. ## Production Metrics and Business Impact The team tracks several operational metrics that reflect both efficiency and adoption. Average Handle Time (AHT) serves as the primary efficiency metric, with initial testing showing a reported 10% reduction. While this is a promising result, it's worth noting that this appears to be from initial tests rather than long-term production data, and the actual sustained impact in full production may vary. Order conversion rate is tracked to ensure the AI assistance isn't negatively impacting sales outcomes. Adoption rate is measured at both the contact level (whether agents use Co-pilot during a conversation) and response level (how often specific suggestions are used), providing insight into how well the system integrates into agent workflows. Edit distance between recommended responses and final sent messages—specifically using Levenshtein Distance—quantifies how much agents modify suggestions. Low edit distances suggest the AI is producing responses close to what agents would write themselves, while high edit distances might indicate quality issues or stylistic mismatches. ## Human-in-the-Loop Design Philosophy A key design decision in this system is the explicit human-in-the-loop approach. Rather than having the LLM directly respond to customers, all suggestions pass through human agents who can accept, modify, or reject them. This provides several benefits from an LLMOps perspective: it creates a natural quality gate, generates valuable training data through agent edits, reduces risk from hallucinations or inappropriate responses, and maintains customer trust through human oversight. This approach is particularly appropriate for customer-facing e-commerce interactions where errors could damage customer relationships or lead to operational problems. It represents a measured approach to deploying generative AI that balances the efficiency gains of automation with the reliability of human judgment. ## Future Development Directions The team outlines two main future development areas. Retrieval Augmented Generation (RAG) is being explored to enhance contextual understanding by connecting the LLM to a database of Wayfair data including product reviews, internal policies, and customer preferences. This would provide real-time access to current information rather than relying solely on what's embedded in prompts, addressing common challenges around knowledge currency and context limitations. Fine-tuning the language model to better match the tone, style, and salesmanship of top-performing agents is also planned. This suggests a move from purely prompt-based control toward model customization, which could improve response quality and consistency while potentially reducing prompt complexity. ## Critical Assessment While the case study presents compelling results, a few considerations warrant attention. The 10% AHT reduction comes from initial tests, and long-term production performance may differ as novelty effects wear off or edge cases emerge. The reliance on LLM-as-judge for quality assessment, while practical, should ideally be validated against human evaluations to ensure alignment. The system's effectiveness likely depends heavily on the quality of policy and product information fed into prompts—keeping this data current and accurate is an ongoing operational challenge not explicitly addressed. Additionally, the human-in-the-loop design, while prudent for quality, means the system amplifies human productivity rather than fully automating responses, which has different scaling characteristics than autonomous systems. Overall, this represents a thoughtful production deployment of generative AI that balances innovation with practical operational considerations, establishing solid foundations for monitoring, evaluation, and iterative improvement.