Digital Ocean developed an inference routing system to address the escalating costs of LLM deployments where frontier models were being used unnecessarily for every task. The solution involves a purpose-built 30-billion parameter routing model that sits as a server-side proxy, intelligently directing requests to appropriate models based on task complexity, real-time latency, cost constraints, and availability metrics. Early customer results demonstrate 40-50% cost reductions with one legal AI startup, with potential savings of up to 80% for some workloads, while maintaining quality and actually reducing latency by approximately 67% compared to using frontier models exclusively.
Digital Ocean has developed and deployed an intelligent inference routing system to address one of the most pressing operational challenges in production LLM deployments: the unsustainable cost of using frontier models for every task. The presenter, Dashi, a Fellow Engineer at Digital Ocean working on data and AI infrastructure, describes how the company built a sophisticated routing layer that sits between clients and model servers to dynamically select appropriate models based on task requirements, cost constraints, and real-time performance metrics.
The motivation for this system stems from widespread industry challenges. Companies like Uber reportedly exhausted their entire annual agentic AI budget within just four months, while retailers like Walmart began implementing caps on AI spending. The fundamental issue is that not every task requires the computational power and expense of frontier models like Claude Opus 5 or the latest GPT series. Classification, labeling, simple summarization, translation, and even many code generation tasks can be adequately handled by smaller or medium-sized models, yet many organizations default to using their most powerful and expensive models for all workloads.
The routing system consists of two primary components working in concert. At its core is a purpose-built 30-billion parameter routing model that was trained specifically for the task of model selection. This specialized model reportedly outperforms general-purpose frontier models on routing-specific benchmarks, demonstrating the value of purpose-built solutions for operational tasks in the LLM stack.
The second component is Plano, an open-source proxy that sits in the middle of the inference pipeline. When a request arrives, Plano forwards it to the routing model, which identifies a candidate set of models capable of handling the task. Plano then performs real-time ranking of these candidates based on current latency metrics, availability status, and cost considerations. This dynamic ranking approach means the system can adapt to changing conditions rather than relying on static configurations or outdated benchmarks.
The entire routing process adds only approximately 200 milliseconds or less of latency, which compares favorably to alternative approaches that might use another LLM to make routing decisions. Such LLM-based routing approaches would impose a “double tax” of both additional cost and latency overhead of around 600 milliseconds, making them impractical for production systems where responsiveness matters.
Digital Ocean highlights a Canadian legal AI startup called Lavo as a customer case study. This service provides legal advice through a combination of AI and human-in-the-loop review. By implementing the intelligent routing system, Lavo achieved 40-50% reduction in inference costs while maintaining output quality. Digital Ocean suggests that some workloads could see cost reductions of up to 80%, though this likely depends heavily on the specific mix of tasks and the extent to which frontier models were being overutilized previously.
Beyond cost savings, the routing system provides operational resilience. If an organization depends on a single model and that model experiences degradation or downtime, the entire application is at risk. By maintaining a portfolio of models and dynamically routing between them, the system provides built-in redundancy and failover capabilities.
The system is designed to minimize operational burden on development teams. All configuration happens server-side using natural language descriptions, eliminating the need for application code changes. Developers don’t need to statically code which model should handle which task or constantly update client-side logic when new models become available or when routing strategies need to change.
Digital Ocean provides preset configurations based on public benchmarks and task-specific evaluations. These presets offer starting points for common tasks like summarization, classification, translation, and code generation, mapping each to appropriate candidate models. However, organizations are expected to customize these configurations based on their specific needs and quality requirements.
A critical aspect of the LLMOps approach described is the emphasis on custom evaluation. The presenter stresses that public leaderboards and benchmarks don’t capture the specific requirements, data distributions, or quality standards of individual organizations. The system includes an evaluation framework that allows teams to plug in their own datasets and continuously validate that the router is making appropriate decisions for their particular use cases.
The evaluation loop is designed to be iterative. Teams deploy the router with initial configurations, run evaluations against their proprietary test sets, identify cases where routing decisions are suboptimal, adjust the configuration, and repeat until performance meets their standards. This approach recognizes that routing quality is context-dependent and cannot be fully determined by generic benchmarks.
Looking forward, Digital Ocean is working on personalization capabilities where the routing system learns from logs and traces to continuously improve its decision-making. This would create a feedback loop where the more an organization uses the system, the better it becomes at making routing decisions tailored to that organization’s specific patterns and requirements, all without requiring active development effort.
An interesting technical consideration mentioned is cache-aware routing, particularly relevant for agentic systems. In agent-based workflows that involve iterative code writing, analysis, and refinement, many prompts are repeated across loop iterations. Models often maintain warm caches for recent prompts, which can significantly improve response times and reduce costs.
If the router changes models mid-session, these cache benefits are lost. The cache-aware routing capability being developed would consider cache state when making routing decisions, potentially preferring to keep requests on the same model when cache hits are valuable, even if another model might otherwise score slightly better on other dimensions. This demonstrates the nuanced considerations required when optimizing production LLM systems beyond simple cost-per-token calculations.
Both the Plano proxy and the routing model are open-sourced, which serves multiple strategic purposes. It allows organizations to host the routing infrastructure themselves if they prefer, avoiding dependency on Digital Ocean’s hosted service. It also enables community contributions and improvements, potentially accelerating development and increasing trust through transparency.
The routing system is designed to integrate with observability systems beyond Digital Ocean’s own infrastructure. While the demo shows integration with Digital Ocean’s inference service, the architecture supports plugging into various backend systems to collect real-time metrics about model performance, availability, and cost.
While the benefits described are compelling, several aspects warrant careful consideration. The claimed 40-50% cost reduction for Lavo and potential 80% savings for other workloads are presented without detailed methodology or baseline information. Organizations evaluating such a system would need to understand their current task distribution and model utilization to estimate realistic savings.
The 30-billion parameter routing model represents a significant resource investment for what is essentially a routing decision. While the 200 millisecond latency is presented as minimal, it’s not zero, and organizations with extremely latency-sensitive applications would need to evaluate whether this overhead is acceptable. The comparison to 600 milliseconds for LLM-based routing is favorable, but static rule-based routing would have even lower latency, though with less flexibility.
The emphasis on purpose-built models for operational tasks is noteworthy and potentially represents an important pattern in LLMOps. Rather than using general-purpose frontier models for every problem, including infrastructure and operational problems like routing, there may be value in specialized models trained for specific operational tasks. However, this approach requires maintaining and updating these specialized models as the landscape of available models changes.
The evaluation framework and iterative refinement process are essential components, but they represent ongoing operational work. Organizations cannot simply deploy the router and forget about it; they must invest in building appropriate test datasets, running evaluations, and refining configurations. This is proper LLMOps practice, but it’s an ongoing commitment rather than a one-time setup.
The demonstration showed a quick deployment process through Digital Ocean’s interface where users can create routers, configure tasks using presets or custom definitions, and immediately test in a playground environment. The ability to compare router performance against specific models like Opus 5 directly in the playground provides immediate feedback on whether the routing strategy is achieving desired cost and latency improvements.
The configuration-driven approach with natural language task definitions represents an interesting abstraction level. Rather than requiring detailed technical specifications, users describe tasks in natural language and the system translates these into routing decisions. This could lower the barrier to adoption, though it may also introduce ambiguity about exactly what routing logic is being applied.
This case study illustrates several important principles for production LLM systems. First, infrastructure and operational concerns like cost optimization and resilience are becoming critical differentiators, not afterthoughts. The rapid emergence of cost optimization as a discipline in the LLM space, compared to the longer evolution of cloud cost optimization, reflects the immediate economic pressures organizations face when deploying these systems at scale.
Second, the routing layer represents a new operational tier in the LLM stack. Just as load balancers and API gateways became standard components in web architecture, intelligent model routing may become a standard layer in LLM infrastructure, mediating between applications and the increasingly diverse landscape of available models.
Third, the tension between frontier model capabilities and practical economic constraints is driving architectural innovation. Rather than simply waiting for models to become cheaper or building applications that require less inference, organizations are developing sophisticated systems to match tasks to appropriately-sized models, fundamentally changing how LLMs are deployed in production.
The open-source approach to critical infrastructure components like routing also suggests an emerging pattern where commercial cloud providers may open-source operational tooling to build ecosystems and trust while monetizing the underlying compute and model access. This could accelerate LLMOps maturity across the industry while still supporting viable business models for infrastructure providers.
Digital Ocean’s inference routing system represents a practical response to real operational challenges in production LLM deployments. By treating model selection as a dynamic optimization problem rather than a static configuration decision, and by building purpose-built infrastructure to solve this problem efficiently, they’ve created a system that addresses cost, latency, and resilience concerns simultaneously. The early results are promising, though organizations considering adoption would need to carefully evaluate the system against their specific workload characteristics and requirements. The emphasis on evaluation, customization, and continuous improvement reflects mature LLMOps thinking, recognizing that generic solutions must be adapted to specific organizational contexts to deliver value.
Notion, a knowledge work platform serving enterprise customers, spent multiple years (2022-2026) iterating through four to five complete rebuilds of their agent infrastructure before shipping Custom Agents to production. The core problem was enabling users to automate complex workflows across their workspaces while maintaining enterprise-grade reliability, security, and cost efficiency. Their solution involved building a sophisticated agent harness with progressive tool disclosure, SQL-like database abstractions, markdown-based interfaces optimized for LLM consumption, and a comprehensive evaluation framework. The result was a production system handling over 100 tools, serving majority-agent traffic for search, and enabling workflows like automated bug triaging, email processing, and meeting notes capture that fundamentally changed how their company and customers operate.
This panel discussion features engineering leaders from Abridge, Replit, and Hebbia discussing their experiences building sophisticated AI agent systems at production scale. Abridge tackles clinical documentation by recording and summarizing doctor-patient conversations for over 250 healthcare systems, addressing challenges around clinical compliance and trust. Replit builds autonomous coding agents that can plan, design, write, test, and debug software with increasingly long-running capabilities. Hebbia creates AI tooling for major financial institutions like KKR and Morgan Stanley, managing extremely spiky workloads with hundreds of thousands of agents processing high-value questions worth hundreds of millions of dollars. All three companies leverage Temporal for durable execution, have moved beyond proof-of-concept to production systems with high stakes, and share common challenges around reliability, cost optimization, model selection, and the evolving balance between agent autonomy and human control.
Zalando, a major e-commerce company, shares their 2.5-year journey implementing agentic engineering and LLMOps practices across 250+ engineering teams. The company built a comprehensive LLM infrastructure starting with a LiteLLM-based API proxy deployed in January 2024, complemented by chat UIs, CLI tools, and coding agents. They addressed challenges around vendor independence, model access, authentication, and governance while implementing risk-based PR approval systems and training programs. The initiative resulted in measurable impacts on PR throughput, code complexity patterns, and developer productivity, with 33% of PRs achieving low-risk auto-approval and significant reductions in PR lead time (20-40% for auto-approved changes).
