Shopify built a continual learning loop for their GraphQL agent (called Sidekick) to address the scaling challenges of using frontier models in production. While frontier models enable rapid initial deployment, they become prohibitively expensive and slow at scale, and critically, they don't learn from production feedback. Shopify's solution—dubbed "the flywheel"—captures production knowledge through carefully defined quality metrics and rubrics, then feeds that knowledge back into model weights through continuous learning. This approach delivered higher quality than frontier models while reducing latency and cutting costs by 96%.
Shopify’s case study on their Sidekick GraphQL agent presents a comprehensive approach to evolving beyond frontier models through continual learning in production environments. The case illustrates a critical transition point that many AI product teams face: frontier models like GPT-4 or Claude enable rapid prototyping and initial deployment with small teams, but as usage scales, the economic and performance characteristics become unsustainable. This case study documents Shopify’s systematic approach to capturing production intelligence and using it to train more efficient, specialized models.
The GraphQL agent represents a concrete application where Shopify processes user queries and generates appropriate GraphQL operations. The success metrics are impressive: matching or exceeding frontier model quality while achieving 96% cost reduction and significant latency improvements. These gains come not from using a different off-the-shelf model, but from implementing what Shopify calls “the flywheel”—a continual learning loop that systematically captures production experience and compresses it into model weights.
The case study opens with a critical observation about the limitations of frontier models in production contexts. While these models excel at general-purpose tasks and enable rapid time-to-market, they suffer from three fundamental limitations when deployed at scale. First, they’re economically inefficient—their general-purpose nature means you’re paying for capabilities you don’t need for your specific use case. Second, they tend to be slower than necessary for specialized tasks. Third, and most importantly from an LLMOps perspective, they operate as static artifacts that don’t improve based on production feedback.
This third limitation is particularly noteworthy. When a user corrects an output, when the system rejects a generated response, or when a recurring failure pattern emerges, frontier models have no mechanism to internalize this feedback. Each production interaction represents “hard-won knowledge” about the product’s specific requirements, user expectations, and edge cases, but this knowledge remains unexploited. The model weights stay frozen at their pre-training state, unable to adapt to the specific domain or learn from mistakes.
Shopify observes that improvements in frontier model deployments typically accumulate in the discrete artifacts surrounding the model: prompt templates get edited, retrieval examples are curated, routing rules are added, and harness code evolves. This creates a situation where production knowledge “piles up in words and code while the model’s weights remain untouched.” This observation captures a key tension in LLMOps: the gap between what the system learns (encoded in prompts and code) and what the model knows (encoded in weights).
Shopify’s answer to this challenge is what they term “the flywheel”—a continual learning loop designed to compress production experience into the continuous space of model weights. This represents a shift from treating models as static components to treating them as continuously evolving systems that improve through production use. The flywheel concept suggests self-reinforcing improvement: better models generate better production data, which trains even better models.
The GraphQL agent serves as Shopify’s “clearest example” of this loop running in production, suggesting they may have other applications where similar approaches are being explored or deployed. The results validate the approach: higher quality outputs compared to frontier models, reduced latency, and a remarkable 96% cost reduction. This cost reduction alone would justify the investment for most organizations operating at scale, but the quality improvements indicate this isn’t simply a cost-saving measure that sacrifices performance.
The case study emphasizes that defining quality is “the most important step in the loop—and the one that teams most often rush.” This observation reflects practical LLMOps wisdom: without clear quality definitions, you cannot effectively evaluate models, cannot create meaningful training signals, and cannot determine whether changes represent improvements. Shopify notes that quality definition begins as a specification but becomes the reward signal that drives learning, making it foundational to the entire system.
The process starts with creating a rubric that translates product requirements into scored criteria. For Shopify’s GraphQL agent, these criteria include completeness, execution quality, response quality, and safety. Each criterion has “concrete anchors” defining what different score levels mean—this prevents annotator drift and ensures consistency. Shopify frames this as “your product team’s definition of good and bad,” positioning it as a cross-functional artifact that bridges product vision and technical implementation.
This rubric becomes “the quality contract for everything downstream,” serving multiple purposes in the LLMOps pipeline. Annotators use it to generate ground truth labels for conversations. Engineers use it to evaluate model variants. The reward modeling system uses it to learn preferences. This single source of truth ensures alignment across the entire pipeline.
A subtle but important detail in Shopify’s approach is their emphasis on sampling randomly from production traffic rather than only curating golden examples. They distinguish between two purposes: “Golden sets test the cases you already know to look for; random samples reveal what good and bad actually look like in production.” This reflects a mature understanding of the gap between anticipated edge cases and real production distribution.
Golden sets are valuable for regression testing and for evaluating performance on known failure modes, but they don’t reveal unknown unknowns. Random production samples capture the true distribution of queries, including cases that product teams haven’t anticipated. This matters for continual learning because you want the model to improve on what users actually do, not just what designers expected them to do.
The inclusion of random samples in ground truth generation ensures that the training signal reflects production reality. This prevents overfitting to curated scenarios and helps the model generalize to the long tail of actual usage. From an LLMOps perspective, this represents good practice in creating training data that reflects deployment conditions.
The case study includes a concrete operational step: once the rubric is defined, have your “two best annotators/product experts blindly annotate 25 random samples and record their inter-annotator agreement.” This step serves as a quality check on the rubric itself. If expert annotators can’t agree on scores, the rubric is ambiguous or incomplete, and downstream systems will learn inconsistent signals.
Using the “two best annotators” rather than average annotators is strategic—you want to test whether the rubric can produce agreement even among those most qualified to judge quality. If it works for experts, it’s more likely to work for the broader annotation team. The blind annotation prevents discussion or consensus-building from masking underlying ambiguity.
The specific number—25 samples—represents a practical minimum for detecting major disagreements without excessive overhead. This is enough to reveal systematic problems with the rubric while remaining feasible as a quick validation step. Though the text cuts off here, the implication is that acceptable inter-annotator agreement validates the rubric and allows teams to proceed with larger-scale annotation.
While the excerpt doesn’t provide complete implementation details, we can infer key architectural components from Shopify’s description. The system must include data collection infrastructure to capture production interactions, including user queries, generated responses, user corrections, rejected outputs, and failure patterns. This requires instrumentation throughout the production system to log relevant events with sufficient context for later learning.
An annotation pipeline processes this production data according to the defined rubric, generating labeled examples. This likely involves a combination of automated labeling (for clear cases), human annotation (for ambiguous cases), and quality assurance processes. The annotation output becomes training data for model improvement.
A training pipeline consumes annotated data to update model weights, presumably through fine-tuning or reinforcement learning approaches. The “reward signal” language and emphasis on quality criteria suggest reward modeling might be part of the approach, though this isn’t explicitly stated. The goal is to “compress production experience into the continuous space of the model’s weights,” implying the knowledge encoded in discrete feedback gets distilled into the model’s parameters.
An evaluation system uses the same quality rubric to assess model variants before deployment, ensuring improvements are real and not artifacts of overfitting to recent data. Model serving infrastructure deploys updated models to production, completing the loop as new production data feeds back into the system.
A key conceptual contribution of this case study is the distinction between knowledge accumulated in “discrete artifacts” versus knowledge compressed into model weights. In typical frontier model deployments, learnings from production manifest as prompt engineering refinements, expanded retrieval databases, more sophisticated routing logic, and defensive code to handle edge cases. These improvements are valuable but have limitations: they increase system complexity, add latency, and don’t fundamentally improve the model’s understanding.
Shopify’s approach internalizes this knowledge into the model itself through continual learning. Instead of adding another prompt instruction or retrieval example, the model learns to handle the case natively. This reduces system complexity and improves efficiency since the model requires less extensive prompting or retrieval context. The trade-off is investment in the continual learning infrastructure and the operational complexity of managing model updates.
From an LLMOps maturity perspective, this represents advancement from static model deployment to dynamic, learning systems. It requires more sophisticated infrastructure but promises better long-term economics and quality for high-volume applications.
The 96% cost reduction is striking but should be interpreted carefully. This likely reflects several factors: smaller, specialized models require fewer computational resources than frontier models; reduced need for extensive prompting and retrieval decreases tokens processed per request; improved quality may reduce retry logic and downstream processing. However, this doesn’t account for the cost of the continual learning infrastructure itself—annotation pipelines, training runs, evaluation systems, and operational overhead.
For organizations at Shopify’s scale, this trade-off clearly favors continual learning. The variable costs of serving every request with frontier models grow linearly with usage, while the fixed costs of continual learning infrastructure can be amortized across high volumes. For smaller-scale applications, the calculus might favor frontier models with simpler deployment patterns.
The latency improvements similarly reflect both model efficiency (smaller models run faster) and reduced complexity (less prompting and retrieval). In e-commerce contexts where user experience is critical, latency improvements directly impact conversion and satisfaction metrics.
Shopify’s case study presents a compelling vision for moving beyond frontier models through continual learning, but several caveats deserve consideration. The approach requires significant upfront and ongoing investment in infrastructure, annotation pipelines, and operational processes. Organizations must have sufficient scale to justify this investment and sufficient AI maturity to implement it effectively.
The emphasis on starting with quality definition is sound advice, but in practice, defining comprehensive quality rubrics that capture all relevant product requirements is challenging. Rubrics inevitably involve trade-offs and may not capture subtle aspects of quality that are hard to articulate but matter to users. Inter-annotator agreement can validate consistency but doesn’t necessarily validate that you’re measuring the right things.
The case study presents results (96% cost reduction, quality improvements) but doesn’t detail the evaluation methodology, making independent verification impossible. These numbers should be viewed as directional indicators rather than guaranteed outcomes for others implementing similar approaches.
The GraphQL agent use case is relatively structured compared to open-ended conversational AI applications. GraphQL queries have well-defined syntax and semantics, making quality assessment more tractable than for creative or subjective tasks. The approach may not transfer directly to all LLM applications.
Finally, continual learning introduces operational challenges not discussed in the excerpt: managing model versions, ensuring backward compatibility, monitoring for quality regressions, handling annotation pipeline failures, and preventing feedback loops where model biases get amplified through production data collection. These are solvable problems but require careful operational practices.
This case study illustrates several important principles for production LLM systems. First, frontier models are valuable starting points but not necessarily optimal end states for high-volume applications. Second, production feedback represents valuable training signal that shouldn’t be wasted. Third, quality definition is foundational to effective LLMOps and deserves substantial investment. Fourth, specialized models trained on domain-specific data can outperform general-purpose models while reducing costs.
The “flywheel” concept provides a useful mental model for continual learning systems: each component (quality definition, annotation, training, deployment, feedback collection) should reinforce the others in a virtuous cycle. Breaking this loop at any point diminishes the system’s ability to improve.
Shopify’s willingness to invest in custom model training and continual learning infrastructure reflects the maturity level of their AI operations. This isn’t a beginner’s approach but rather represents advanced LLMOps practice suitable for organizations with scale, resources, and technical sophistication. For others, the lessons about quality definition and production feedback collection remain valuable even if full continual learning isn’t immediately feasible.
This panel discussion features experts from Microsoft, Google Cloud, InWorld AI, and Brazilian e-commerce company IUD (Prosus partner) discussing the challenges of deploying reliable AI agents for e-commerce at scale. The panelists share production experiences ranging from Google Cloud's support ticket routing agent that improved policy adherence from 45% to 90% using DPO adapters, to Microsoft's shift away from prompt engineering toward post-training methods for all Copilot models, to InWorld AI's voice agent architecture optimization through cascading models, and IUD's struggles with personalization balance in their multi-channel shopping agent. Key challenges identified include model localization for UI elements, cost efficiency, real-time voice adaptation, and finding the right balance between automation and user control in commerce experiences.
Shopify built a fine-tuned tool-calling agent based on Qwen3-32B to generate Flow automation workflows from natural language queries within their Sidekick AI assistant. The team addressed the cold-start problem by reverse-engineering synthetic training data from existing production workflows, then improved model performance by translating their JSON DSL into Python for training. The resulting model is 2.2x faster and 68% cheaper than the frontier model it replaced, though initial deployment revealed a 35% gap in activation rates that was closed through a weekly retraining flywheel incorporating real merchant data, LLM-based evaluation judges, and continuous improvement loops.
Adaptive ML addresses the challenge that 95% of GenAI pilots fail to reach production by advocating for reinforcement learning as the core post-training technique. The company argues that MVP solutions built on proprietary models or instruction fine-tuning lack systematic improvement mechanisms, whereas RL enables continuous integration of feedback from production environments. Their RLOps platform serves enterprises like AT&T, Manulife, and CCS Medical Supply, enabling them to train smaller, faster, and more cost-effective specialized LLMs. The approach particularly excels for agentic use cases, where RL's ability to train models in simulated environments with business-specific rewards unlocks production-grade performance while reducing inference costs by millions of dollars through model compression.
