## Overview Amazon's Catalog Team presents a sophisticated LLMOps case study focused on building a self-learning generative AI system for product catalog enrichment at massive scale. The Amazon.com Catalog serves as the foundation for customer shopping experiences, requiring extraction of structured attributes (dimensions, materials, compatibility, technical specifications) and generation of optimized content like product titles from millions of daily seller submissions. The challenge was fundamentally one of production-scale LLMOps: how to continuously improve model performance across millions of products while managing computational costs and avoiding the unsustainable manual work of having applied scientists constantly analyze failures, update prompts, test changes, and redeploy. The team's solution represents an innovative approach to LLMOps that moves beyond traditional model selection toward building systems that accumulate domain knowledge through actual production usage. Rather than choosing between large accurate models or efficient smaller models, they developed a multi-model architecture where disagreements between models become learning signals that drive continuous improvement. ## Core Technical Architecture The fundamental insight underlying this LLMOps architecture came from treating model disagreements as features rather than bugs. When the team deployed multiple smaller models to process the same products, they discovered that disagreements correlated strongly with cases requiring additional investigation. This led to a three-tier architecture built on Amazon Web Services infrastructure: **Worker Layer**: Multiple lightweight models operate in parallel using a generator-evaluator pattern. Generators extract product attributes while evaluators assess those extractions. This creates productive tension conceptually similar to GANs but operating at inference time through prompting rather than training. Workers are explicitly prompted to be critical and adversarial, with evaluators instructed to scrutinize extractions for ambiguities and potential misinterpretations. The team implements workers both as models accessed through Amazon Bedrock (such as Amazon Nova Lite) and as open-source models deployed on Amazon EC2 GPU instances for greater cost control at scale. Workers are designed to be non-agentic with fixed inputs, making them batch-friendly and highly scalable. **Supervisor Layer**: When workers disagree, the system invokes a supervisor agent built on more capable models like Anthropic Claude Sonnet accessed through Amazon Bedrock. The supervisor doesn't simply resolve disputes—it investigates why disagreements occurred, determines what context or reasoning workers lacked, and generates reusable learnings. The supervisor is implemented as an agent with access to specialized tools for deeper investigation, capable of pulling in additional signals like customer reviews, return reasons, and seller history that would be impractical to retrieve for every product. This asymmetry between lightweight workers and capable supervisors is crucial for efficiency: routine cases are handled through consensus at minimal cost, while expensive supervisor calls are reserved for high-value learning opportunities. **Knowledge Base Layer**: Learnings extracted by the supervisor are stored in a hierarchical knowledge base implemented with Amazon DynamoDB. An LLM-based memory manager navigates this knowledge tree to place each learning appropriately, starting from the root and traversing categories and subcategories. The manager decides at each level whether to continue down an existing path, create a new branch, merge with existing knowledge, or replace outdated information. During inference, workers receive relevant learnings in their prompts based on product category, automatically incorporating domain knowledge from past disagreements without requiring model retraining. The production system is orchestrated using Amazon Bedrock AgentCore, which provides runtime scalability, memory management, and observability for deploying self-learning systems reliably at scale. Product data flows through generator-evaluator workers, with agreements stored directly and disagreements routed to the supervisor. A learning aggregator synthesizes insights, adapting aggregation strategy to context: high-volume patterns get synthesized into broader learnings while unique or critical cases are preserved individually. Human review queues managed through Amazon SQS and observability through Amazon CloudWatch complete the production architecture. ## The Self-Learning Mechanism What makes this a genuinely self-learning system rather than just a clever routing mechanism is how it handles multiple sources of disagreement signals and converts them into accumulated institutional knowledge: **Inference-Time Learning**: When workers disagree on attribute extraction—for example, interpreting the same technical specification differently—this surfaces cases requiring investigation. The team discovered a "sweet spot" for disagreement rates: moderate rates yield the richest learnings (high enough to surface meaningful patterns, low enough to indicate solvable ambiguity). When disagreement rates are too low, they typically reflect noise or fundamental model limitations; when too high, it signals that worker models or prompts aren't yet mature enough, triggering excessive supervisor calls that undermine efficiency gains. **Post-Inference Learning**: The system also captures feedback signals after initial processing. Sellers express disagreement through listing updates and appeals, indicating original extractions might have missed context. Customers disagree through returns and negative reviews, often indicating product information didn't match expectations. These post-inference human signals feed into the same learning pipeline, with the supervisor investigating patterns and generating learnings that prevent similar issues across future products. **Learning Propagation**: The critical innovation is how learnings become immediately actionable. When the supervisor investigates a disagreement—say, about usage classification based on technical terms—it might discover that those terms alone were insufficient and that visual context and other indicators needed to be considered together. This learning immediately updates the knowledge base, and when injected into worker prompts for similar products, helps prevent future disagreements across thousands of items. No retraining required; the system evolves through prompt engineering and knowledge augmentation. ## LLMOps Best Practices and Lessons The case study offers valuable insights for teams implementing similar production LLM systems, though readers should note these represent Amazon's experience and may require validation in other contexts: **When This Architecture Works**: The team identifies high-volume inference with input diversity as ideal, where compounded learning creates value. Quality-critical applications benefit from consensus-based quality assurance. Evolving domains with constantly emerging patterns and terminology see particular value. However, the architecture is less suitable for low-volume scenarios (insufficient disagreements for learning) or use cases with fixed, unchanging rules. **Critical Success Factors**: Defining disagreements appropriately is fundamental. With generator-evaluator pairs, disagreement occurs when the evaluator flags extractions as needing improvement. The key is maintaining productive tension between workers—if disagreement rates fall outside the productive range, teams should consider more capable workers or refined prompts. Tracking learning effectiveness through declining disagreement rates over time serves as the primary health metric. If rates stay flat, teams should examine knowledge retrieval, prompt injection, or evaluator criticality. Knowledge organization must be hierarchical and actionable—abstract guidance doesn't help; specific, concrete learnings directly improve future inferences. **Common Pitfalls**: The team warns against focusing on cost reduction over intelligence—cost reduction is a byproduct, not the goal. "Rubber-stamp evaluators" that simply approve generator outputs won't surface meaningful disagreements; they must be prompted to actively challenge extractions. Poor learning extraction where supervisors only fix individual cases rather than identifying generalizable patterns undermines the architecture. Knowledge rot occurs without proper organization, making learnings unsearchable and unusable. ## Deployment Strategies The team outlines two approaches for production deployment, both using the same underlying architecture: **Learn-Then-Deploy**: Start with basic prompts and let the system learn aggressively in a pre-production environment. Domain experts audit the knowledge base (not individual outputs) to ensure learned patterns align with desired outcomes before deploying with validated learnings. This works well for new use cases where teams don't yet know what "good" looks like—disagreements help discover the right patterns, and knowledge base auditing shapes them before production. **Deploy-and-Learn**: Start with refined prompts and good initial quality, then continuously improve through ongoing learning in production. This suits well-understood use cases where teams can define quality upfront but want to capture domain-specific nuances over time. ## Production Considerations and Tradeoffs While the case study presents impressive results, readers should consider several aspects critically: **Complexity vs. Benefit Tradeoffs**: This architecture introduces significant operational complexity compared to single-model approaches. Teams must manage multiple worker models, maintain supervisor agents with tool integrations, operate a dynamic knowledge base with LLM-based memory management, and orchestrate agreement/disagreement routing. The benefits—declining costs and improving accuracy—accrue over time, but upfront engineering investment is substantial. Organizations should carefully assess whether their scale and use case justify this complexity. **Knowledge Base Maintenance**: The hierarchical knowledge base represents a critical component requiring ongoing governance. As learnings accumulate, organizations need processes for auditing knowledge quality, resolving conflicts between learnings, deprecating outdated information, and ensuring learnings generalize appropriately without overfitting to specific cases. The case study mentions domain experts can directly contribute by adding or refining entries, but this implies ongoing maintenance overhead. **Disagreement Rate Dynamics**: The "sweet spot" for disagreement rates is task and domain-specific, requiring empirical tuning. Teams must establish monitoring to detect when disagreement rates fall outside productive ranges and have processes for adjusting worker models, prompts, or supervisor invocation thresholds. The architecture's efficiency gains depend on declining disagreement rates over time—if learnings don't effectively prevent future disagreements, the system degenerates into expensive supervisor calls without the promised benefits. **Observability Requirements**: The case study mentions Amazon Bedrock AgentCore Observability for tracking which learnings drive impact, but production systems need comprehensive instrumentation beyond this. Teams must monitor worker agreement/disagreement rates by product category, supervisor invocation frequency and latency, learning effectiveness metrics, knowledge base growth and utilization, and end-to-end system costs. Without detailed observability, teams cannot validate the architecture is delivering promised benefits. **Human-in-the-Loop Considerations**: The architecture incorporates human signals (seller updates, customer returns) but doesn't detail how false positives are handled—situations where the AI was correct but humans disagree for other reasons. Production systems need mechanisms to distinguish legitimate disagreements from noise in human feedback signals to avoid contaminating the knowledge base with incorrect learnings. ## Results and Impact The case study reports that error rates fell continuously through accumulated learnings from resolved disagreements, though specific quantitative metrics are not provided. The team emphasizes that improvement came without retraining, purely through learnings stored in the knowledge base and injected into worker prompts. The system evolved from generic understanding to domain-specific expertise, learning industry-specific terminology, discovering contextual rules varying across categories, and adapting to requirements no pre-trained model would encounter. Cost efficiency improved as the architecture design inherently creates a virtuous cycle: as learnings accumulate and disagreement rates drop, supervisor calls naturally decline, reducing computational costs while quality improves. This contrasts with traditional approaches where quality and cost are typically in tension. The traceability introduced by the knowledge base shifts auditing from reviewing samples of millions of outputs (where human effort grows proportionally with scale) to auditing the knowledge base itself, which remains relatively fixed in size regardless of inference volume. ## Technical Implementation Details The reference architecture leverages several AWS services in production: Amazon Bedrock provides access to diverse foundation models enabling deployment of different models as workers and supervisors. Amazon EC2 GPU instances offer full control over worker model selection and batch throughput optimization for open-source models. Bedrock AgentCore supplies the runtime for supervisor agents with specialized tools and dynamic knowledge base access. Amazon DynamoDB stores the hierarchical knowledge base. Amazon SQS manages human review queues. Amazon CloudWatch provides observability into system performance. The generator-evaluator pattern deserves particular attention as a production LLMOps technique. By creating adversarial tension through prompting rather than architecture, the team surfaces disagreements representing genuine complexity rather than letting ambiguous cases pass through. Evaluators are explicitly instructed to be critical, scrutinizing extractions for ambiguities, missing context, or potential misinterpretations. This inference-time adversarial dynamic is more flexible than training-time approaches, allowing rapid adjustment through prompt engineering. ## Broader Implications for LLMOps This case study represents a significant contribution to production LLMOps patterns beyond catalog enrichment. The fundamental insight—treating disagreements as learning signals and building systems that accumulate domain knowledge through usage—applies broadly to high-volume AI applications. The architecture shifts the question from "which model should we use?" to "how can we build systems that learn our specific patterns?" However, organizations should approach implementation with realistic expectations. This is not a simple pattern to implement—it requires sophisticated orchestration, careful tuning of disagreement thresholds, robust knowledge management processes, and comprehensive observability. The benefits accrue over time through accumulated learnings, meaning organizations need patience and commitment to see results. The case study is also clearly promotional for AWS services, though the underlying architectural patterns could potentially be implemented with other infrastructure. The emphasis on knowledge base auditing as a scaling strategy for quality assurance is particularly noteworthy for production LLMOps. Rather than sampling outputs (which doesn't scale), auditing the knowledge base that drives behavior creates a more manageable quality control surface. This shifts AI governance from reacting to errors to proactively shaping system behavior through knowledge curation—a more sustainable approach for production systems processing millions of inferences. Overall, this case study demonstrates mature LLMOps thinking: building systems that improve through production usage, managing cost-quality tradeoffs through architecture rather than just model selection, creating governance mechanisms that scale with inference volume, and treating production deployment as an opportunity to accumulate institutional knowledge rather than a one-time model deployment event. While implementation challenges are real and results may vary by context, the patterns described offer valuable insights for organizations building production LLM systems at scale.