This panel discussion brings together leaders from NVIDIA, Cognition, and OpenRouter to explore the emerging practice of model routing in production AI systems. The participants discuss how organizations are moving beyond single-model deployments to multi-model architectures that intelligently route tasks between frontier models and smaller specialized models. The conversation covers Cognition's Devin Fusion system that achieves 40% cost reduction while maintaining or exceeding frontier model performance, OpenRouter's marketplace approach to model routing, and NVIDIA's work on co-designing models with orchestration systems. Key challenges addressed include context sharing between models, KV cache optimization, detecting when models are out of their depth, and training models to collaborate effectively with other models.
This panel discussion provides a comprehensive look at one of the most critical emerging patterns in production LLMOps: model routing and multi-model orchestration. The conversation features perspectives from three key players - NVIDIA (representing infrastructure and model development), Cognition (representing production AI agent deployment with their Devin product), and OpenRouter (representing model marketplace and routing infrastructure). The discussion reveals that model routing has evolved from a niche optimization technique to a fundamental architecture pattern for production AI systems, driven by the intersection of cost pressures, capability differentiation across models, and the rise of agentic workflows.
The panelists establish that model routing addresses multiple critical production concerns simultaneously. Walden from Cognition frames the problem from their customer perspective: organizations deploying AI systems constantly ask about ROI and which tasks justify expensive frontier models versus cost-efficient alternatives. This isn’t merely about saving money - it’s about making AI deployment economically sustainable at scale.
Carter from NVIDIA emphasizes that as frontier models become more capable, they also become more expensive, creating a cost barrier that makes these tools “somewhat cost prohibitive” for individual developers, startups, and small companies. The challenge becomes delivering the same desired outputs while maintaining economic viability. This economic pressure is particularly acute for agentic workflows that generate tokens continuously, fundamentally changing the cost profile compared to traditional request-response patterns.
Alex from OpenRouter provides concrete evidence of this evolution. Their auto-router feature existed for nearly two years with minimal adoption because users wanted specific models. The inflection point came in January when adoption exploded, driven by applications like OpenClaw that exposed the economic inefficiency of routing all tasks to expensive models. OpenClaw’s heartbeat mechanism that pings every ten minutes revealed a fundamental truth: production AI systems have wildly varying intelligence requirements across different tasks and phases of operation.
Cognition’s Devin Fusion represents one of the most sophisticated production implementations of model routing discussed. Walden makes the striking claim that their system achieves better than frontier model performance while reducing costs by 40% - a counterintuitive result that challenges assumptions about the performance-cost tradeoff.
The key architectural innovation is what Cognition calls a “sidekick” pattern rather than traditional sub-agents. The distinction is critical for production efficiency. In a traditional sub-agent architecture, context gets fragmented and must be repeatedly provided to different agents, multiplying token costs and losing KV cache benefits. The sidekick pattern maintains a continuous running context for the smaller model, preserving cache hits and reducing costs by 10x on cached tokens.
The system architecture positions a frontier model as the orchestrator that handles planning and hard decision-making while delegating implementation work to smaller, more cost-efficient models. Walden emphasizes an important nuance: smarter models are actually better at delegation, creating a virtuous cycle where intelligence improves the overall system efficiency. This challenges the naive approach of simply routing “easy” tasks to small models and “hard” tasks to large models.
The delegation strategy enables an interesting capability expansion. Because smaller models are significantly cheaper per token, the system can afford to use many more tokens from the smaller model while staying within the budget that a frontier model call would have consumed. This allows spawning multiple specialized exploration agents that might examine a codebase more comprehensively than a single frontier model with limited context could achieve.
Cognition has also identified that coding domains exhibit “jagged capabilities” where different models excel at different sub-tasks based on their training corpora. A model might be superior at matplotlib visualization but weaker at scikit-learn tasks. Understanding these capability profiles at an intimate level allows for sophisticated routing that leverages the strengths of each model in the pool.
Context sharing between models emerges as one of the most complex technical challenges in multi-model systems. When a sidekick agent completes work and needs to escalate to the main model, the system must balance information completeness against token efficiency. Dumping the entire execution trace to the frontier model would be prohibitively expensive and defeat the cost optimization goals.
Cognition’s approach applies context compaction techniques already developed for long-running agents. Rather than passing full file contents, the sidekick might reference files by name, allowing the more token-efficient frontier model to selectively read only the portions it needs. The frontier models demonstrate superior tool usage efficiency, often able to verify work completion with a single command rather than exhaustive inspection.
The panelists discuss several technical approaches to context compaction. One involves training models to present compacted context back to orchestrator models. Another uses abstract syntax trees and structured representations rather than raw text, which provides more lossless compression that can retain agent state. The challenge is balancing compression against information loss - as one panelist notes, compaction is inherently lossy, requiring careful engineering to determine what can be safely compressed.
KV cache management adds another dimension to context challenges. Most API providers evict caches after around 5 minutes of inactivity due to the cost of keeping them loaded in GPU memory. This creates a natural refresh cycle that Cognition exploits - when refreshing cache anyway, they get a “free” frontier model call to check if the smaller model has gone down a rabbit hole. For self-hosted deployments, organizations can tune cache duration based on their specific workload characteristics and memory availability, potentially achieving much longer cache retention.
OpenRouter provides a fascinating perspective as both a model marketplace and routing infrastructure provider. Their platform abstracts the complexity of interfacing with multiple model providers while enabling sophisticated routing strategies. Alex explains they’ve evolved from a simple auto-router discovery tool to offering multiple routing products addressing different use cases.
Their Pareto code router provides Pareto optimal model selection for coding tasks given tunable thresholds, allowing developers to explicitly control the performance-cost tradeoff. The fusion product orchestrates multiple models to produce fused results, similar to Cognition’s approach. The key insight is providing both simple “set a slug and forget” interfaces for easy adoption alongside sophisticated primitives for advanced orchestration.
A critical debate within OpenRouter concerns whether the outer orchestration model should be the large or small model. The choice has non-obvious cost implications because large model caching can actually provide dramatic price savings compared to small model caching. The optimal choice depends heavily on task characteristics and whether prompts are in-distribution or out-of-distribution for the model’s training data.
OpenRouter has observed that small models can actually increase costs on out-of-distribution tasks. On terminal bench evaluations, Opus performs three times better at one-tenth the cost of Haiku for certain tasks, despite Haiku being significantly cheaper per token. This occurs because small models thrash with excessive tool calls and loop iterations when facing tasks beyond their training. For in-distribution tasks like simple text classification, small models provide clear cost advantages.
The platform passes through cache hits directly to users and is developing functionality to intelligently switch models even mid-cache when significant benefit exists, allowing users to tune tolerance for losing remaining cache duration. This demonstrates the intricate optimization landscape where cache economics interact with model routing decisions.
A frontier area discussed is training models specifically to collaborate with other models, rather than treating models as fixed artifacts to orchestrate. Walden describes Cognition’s work reinforcement learning models in two configurations: as orchestrators deciding what to delegate to other models, and as executors following instructions from other models. This co-design of models with orchestration systems represents a significant evolution from naive model routing.
NVIDIA’s work includes NeMo Triton models released with complete datasets, weights, and recipes for customization, acknowledging that model customization for specific orchestration patterns will become crucial. They’re exploring FlexRun technology that distills main models into smaller footprints and switches which weights activate based on task complexity. This allows flexible model size within a single artifact based on the detected task requirements.
The discussion touches on using model internal states for routing decisions. Hallucination probes can analyze internal model states through magnitude analysis or linear probes to detect when a model is becoming uncertain or “lost” in its reasoning. This provides signals beyond simple token generation patterns for determining when escalation to a more capable model is needed. The KV cache itself is fundamentally a collection of vectors that can be classified to understand model state.
The panelists highlight that cost economics change dramatically between self-hosted models and API providers. Self-hosting provides control over cache duration, memory allocation, and workload-specific optimization. While API providers must amortize across diverse use cases with general optimizations, self-hosting allows tuning for specific workload shapes - whether that’s 32K average cache with 1K input/output or different profiles.
Walden reveals that Cognition started by buying direct compute capacity from providers rather than paying per-token, recognizing that compute economics for cached tokens were far cheaper than token-based pricing before providers offered explicit cache pricing. This enabled building the first production agents when they would have been prohibitively expensive on standard pricing.
For organizations with on-premises compute like DGX systems, model routing serves an additional purpose: maximizing utilization of already-purchased hardware. When a DGX isn’t at full utilization, routing appropriate workloads locally rather than to cloud APIs maximizes the value of the fixed hardware investment, with only power costs as the marginal expense.
Carter from NVIDIA introduces another routing dimension: privacy-sensitive workload detection. Systems can identify when prompts contain sensitive information and route those to on-device or local models, while potentially anonymizing information for more advanced cloud processing. This creates hybrid architectures that balance privacy protection, cost optimization, and capability requirements.
This pattern is particularly relevant for enterprise deployments where data governance requires certain processing to remain local while still leveraging frontier model capabilities for less sensitive operations. The routing logic must understand both technical requirements and policy constraints.
The discussion reveals several practical challenges that production deployments face. Prompt portability between models is limited because different models and architectures respond differently to the same prompts. This requires careful prompt engineering for each model in the routing pool, which the panelists frame as part of the startup building process for agent companies. The value of companies like Cognition includes accumulated knowledge of “doom loops” and failure patterns across different models with recovery strategies.
Cognition is developing auto-research systems that continuously learn from production usage. When users manually upgrade or downgrade models, or when the system detects a routing error, these signals feed back into routing optimization. This creates a flywheel where production data continually improves routing decisions without manual intervention.
The panelists discuss prompt tuning approaches, with some skepticism toward mechanical gradient-descent-based frameworks from a few years ago. The preferred approach uses frontier model intelligence to analyze routing decisions in context, identify failures, and suggest prompt improvements that can be regression tested. This human-in-the-loop approach with AI assistance is deemed more trustworthy than fully automated mechanical tuning.
The conversation concludes with reflection on whether routing will be a standalone product category or become integrated infrastructure. The consensus is that routing will be distributed across both improved model capabilities and harness sophistication. Models are already becoming naturally more collaborative and better at delegation, as evidenced by newer frontier models.
However, the non-deterministic nature of models creates a low-trust environment requiring controller-based arbitration at the orchestration level. The analogy to web traffic routing is raised - while individual services improved, centralized routing controls emerged over time as the optimal architecture pattern. The panelists expect similar evolution in model routing.
Cache economics will continue driving multi-model architectures even if individual models become more efficient. Models have limited memory and context windows that degrade beyond certain thresholds (with recommendations to stay under 100-200K tokens despite million-token advertised windows). Context limitations and cache hit economics will continue favoring architectures that maintain context in appropriate models rather than consolidating everything in one model.
The discussion reveals a maturing field moving from research to production deployment with sophisticated patterns emerging. The technical challenges around context management, cache optimization, capability profiling, and collaborative training are being actively addressed by industry leaders. The economic imperative combined with technical capability is driving rapid evolution in this space, with expectations that current techniques will seem primitive within a year as the field advances.
This case study examines Cursor's implementation of reinforcement learning (RL) for training coding models and agents in production environments. The team discusses the unique challenges of applying RL to code generation compared to other domains like mathematics, including handling larger action spaces, multi-step tool calling processes, and developing reward signals that capture real-world usage patterns. They explore various technical approaches including test-based rewards, process reward models, and infrastructure optimizations for handling long context windows and high-throughput inference during RL training, while working toward more human-centric evaluation metrics beyond traditional test coverage.
Shopify's CTO discusses how the company has achieved near-universal AI adoption internally, with nearly 100% of employees using AI tools daily as of December 2025. The company has developed sophisticated internal platforms including Tangle (an ML experimentation framework), Tangent (an auto-research loop for automatic optimization), and SimGym (a customer simulation platform using historical data). These systems have enabled dramatic productivity improvements including 30% month-over-month PR merge growth, significant code quality improvements through critique loops, and the ability to run hundreds of automated experiments. The company provides unlimited token budgets to employees and emphasizes quality token usage over quantity, focusing on efficient agent architectures with critique loops rather than many parallel agents. They've also implemented Liquid AI models for low-latency applications, achieving 30-millisecond response times for search queries.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
