## Overview Gradient Labs operates an AI agent designed specifically for customer interactions in the financial services sector, where reliability is paramount. The company's case study offers a detailed look at the production infrastructure challenges of running agentic systems that chain multiple LLM calls together, each with associated costs and latency implications. Their approach centers on building resilience through multi-provider failover strategies, durable execution frameworks, and sophisticated monitoring systems. The context is particularly demanding: when customers contact their bank about money-related issues, there is zero tolerance for system unavailability. The company uses a blend of different LLMs under the hood to construct high-quality answers, making LLM availability a critical dependency. Their architecture demonstrates a mature approach to LLMOps that goes beyond simple API calls to encompass provider diversity, model redundancy, and intelligent traffic management. This case study is particularly valuable because it addresses real production challenges that emerge when running long-duration agentic workflows at scale in a high-stakes industry. ## Architectural Paradigm Shift Gradient Labs highlights a fundamental difference between traditional server-client architectures and agentic systems. In conventional request-response patterns, requests complete within a few hundred milliseconds, and failures typically trigger a complete retry of the entire request. However, agentic systems involve chains of LLM calls that can span much longer durations. Each individual LLM call carries both user-facing latency costs and monetary costs, making it inefficient and expensive to retry entire request chains when only a single step fails. The company identifies two naive approaches to solving this problem: manually writing state to a database after each step to create recovery checkpoints (which introduces complexity around ensuring database writes succeed), or implementing retry logic at every step of the agent (which conflates business logic with resilience concerns). Instead, Gradient Labs adopted Temporal, a durable execution system that provides automatic checkpointing out of the box. This architectural choice separates the concerns of agent logic from failure recovery, allowing the system to resume from the last successful step rather than restarting entire workflows. This represents a thoughtful approach to a real LLMOps challenge. While the blog post positions this as solving a clear problem, it's worth noting that durable execution systems like Temporal add operational complexity and require teams to understand their programming model. The tradeoff appears justified for Gradient Labs' use case given the financial services context, but teams in lower-stakes environments might find simpler retry mechanisms sufficient. ## Multi-Provider Architecture A core design principle at Gradient Labs is maintaining flexibility to experiment with, evaluate, and adopt the best LLMs for each component of their agent. They currently use three major model families: OpenAI models (served via OpenAI and Azure APIs), Anthropic models (served via Anthropic, AWS, and GCP APIs), and Google models (served via GCP APIs in different regions). This multi-provider strategy serves two primary purposes: spreading traffic across providers to maximize utilization of per-provider rate limits, and enabling failover when encountering errors, rate limits, or latency spikes. The system implements an ordered preference list for each completion request. For example, a GPT-4.1 request might have preferences ordered as (1) OpenAI, (2) Azure. These preferences can be configured both globally and on a per-company basis, with proportional traffic splitting to distribute load according to desired ratios. When certain error conditions arise, the system automatically fails over to the next provider in the preference list. This architecture demonstrates sophisticated production thinking around LLM infrastructure. By maintaining multiple paths to the same model families, Gradient Labs reduces dependency on any single provider's uptime or capacity. However, this approach also introduces significant operational complexity—teams must manage API keys, monitor rate limits, and track performance across multiple providers. The blog post doesn't detail the engineering effort required to build and maintain this infrastructure, which is likely substantial. Organizations evaluating this pattern should consider whether the reliability benefits outweigh the operational overhead for their specific use case. ## Failover Decision Logic The nuance of any failover system lies in determining when to fail over versus when to handle errors differently. Gradient Labs identifies four categories of responses that require distinct handling strategies: Successful but invalid responses represent cases where the LLM generates output that doesn't match expected formatting—for example, when the system requests a decision within specific XML tags but the response omits them. The company explicitly does not fail over for these cases, recognizing that the underlying API is functioning correctly even if the model output requires different handling (likely retry with prompt adjustments or parsing fallbacks). For error responses, particularly 5XX server errors from LLM APIs, the system initiates failover to alternative providers. This is standard practice for distributed systems, treating LLM providers as potentially unreliable dependencies that require redundancy. Rate limiting receives special treatment: when a request fails due to rate limits, the system not only fails over to an alternative provider but also marks the rate-limited provider as "unavailable" in a cache for a short duration. This optimization prevents wasting latency on subsequent requests to a resource that's already over capacity. This is a particularly clever detail that demonstrates production maturity—the system learns from rate limit signals and proactively avoids constrained resources rather than repeatedly hitting them. Latency-based failover represents the most sophisticated category. The system monitors request duration and fails over when individual requests exceed a timeout in the p99+ percentile of latency. This catches scenarios where specific requests are abnormally slow, which could indicate provider issues or model-specific problems. However, as the case study later reveals, this approach has limitations that the team continues to refine. The categorization demonstrates thoughtful consideration of different failure modes. However, the blog post doesn't specify which 5XX errors trigger failover and which don't—some 5XX errors might be request-specific rather than indicating provider issues. Similarly, the cache duration for marking providers unavailable after rate limiting isn't specified, which would be an important tuning parameter. These details matter significantly in production but are abstracted in the presentation. ## Model-Level Failover Beyond provider failover, Gradient Labs implements model-level failover for catastrophic scenarios where an entire model family becomes unavailable across all providers. For example, if Google experiences a complete outage, all Gemini model requests would fail regardless of which API endpoint (GCP in different regions) is attempted. In these rare cases, the system can switch to a completely different model family. The primary challenge with model failover is prompt compatibility—prompts optimized for one model don't necessarily perform well with others. Gradient Labs addresses this by designing and evaluating multiple prompt-model pairs as part of their development lifecycle. For critical system components, they maintain tailored prompts for both primary and backup models. This approach provides two benefits: protection against complete model family outages, and the ability to fail over from newer experimental models (which often have lower rate limits) to older, more established models with higher capacity allocations. This represents a significant engineering investment that many organizations might overlook. Maintaining multiple prompt versions, evaluating their performance across different models, and keeping them synchronized with system changes requires substantial ongoing effort. The blog post positions this as already part of their development lifecycle, suggesting they've integrated prompt versioning into their standard practices rather than treating it as an additional burden. However, this also means that adding new capabilities to the agent requires designing and testing prompts across multiple model families, potentially slowing down feature development. The model failover strategy also reveals an interesting tension in LLMOps: newer, more capable models often come with stricter rate limits and lower availability guarantees, while older models offer higher capacity but potentially lower quality. Gradient Labs' architecture allows them to prefer newer models while maintaining reliable fallbacks, but this creates a two-tier system where some percentage of requests receive responses from older, presumably less capable models. The blog post doesn't discuss how they measure or manage the quality implications of these failovers, which would be important for understanding the full production impact. ## Continuous Improvement: Latency Distribution Shifts The case study includes a concrete example of system evolution based on production incidents. The existing failover mechanism protected against individual requests taking too long by timing out and failing over to alternative providers. These timeouts target abnormally slow requests, typically in the p99 percentile of latency. However, the team encountered a scenario where the entire latency distribution shifted rather than just outliers becoming slower. During one incident with a provider, mean latency spiked and the p75+ latency exceeded 10 seconds. This increased overall agent latency significantly but didn't trigger the existing failover mechanism because individual requests remained within their p99 timeout thresholds—the timeout values themselves were calibrated for a different latency distribution. The team detected this through latency-based alerts and manually invoked failover, but the incident revealed a gap in their automatic resilience systems. This example demonstrates honest and valuable production learning. Many case studies present polished final solutions, but Gradient Labs shares an ongoing challenge where their existing approach had limitations. The question they pose—whether to implement automatic failover when observing abnormal shifts in latency distributions—represents a sophisticated next step that would require statistical monitoring of latency patterns rather than simple threshold-based alerts. However, implementing distribution-shift detection introduces new complexities: determining what constitutes an "abnormal" shift, avoiding false positives from normal traffic variations, and deciding when to fail back to the original provider once latency normalizes. The blog post doesn't commit to a specific solution, suggesting this remains an open area of development. This kind of transparency about ongoing challenges is valuable for the LLMOps community, as it highlights real production problems that don't have simple answers. ## LLMOps Maturity and Tradeoffs The Gradient Labs architecture demonstrates significant LLMOps maturity across several dimensions. Their multi-provider strategy with configurable preferences and proportional traffic splitting shows sophisticated production infrastructure. The integration of Temporal for durable execution addresses a real challenge in agentic workflows. The multi-level failover (provider and model) with tailored prompts represents substantial engineering investment. The continuous monitoring with latency-based alerts and willingness to evolve the system based on incidents indicates a learning organization. However, the case study also implicitly reveals significant tradeoffs. The operational complexity of managing multiple providers, APIs, and model versions is substantial. The engineering effort required to maintain prompt variants across models and incorporate this into the development lifecycle slows down iteration. The need for sophisticated monitoring, caching strategies for rate-limited providers, and distribution-shift detection requires dedicated infrastructure and expertise. The system's reliance on Temporal adds a complex dependency that requires operational expertise to run reliably. For organizations evaluating similar approaches, the key question is whether their reliability requirements justify this complexity. In financial services where customer trust and regulatory compliance are paramount, Gradient Labs' investment appears well-justified. However, teams in less critical domains might find that simpler approaches—perhaps using a single provider with basic retry logic—provide adequate reliability with far less operational overhead. The case study also doesn't address several important production concerns. There's no discussion of cost management across multiple providers or how they optimize spending while maintaining reliability. The latency impact of failover attempts isn't quantified—how much additional latency do customers experience when the system tries multiple providers? How do they balance failing over quickly to improve reliability versus waiting longer to avoid unnecessary failovers? The blog post doesn't mention how they evaluate whether backup models produce acceptable quality when failover occurs, which seems critical for a financial services application. ## Evaluation and Testing While the case study doesn't explicitly detail evaluation practices, several elements suggest a mature approach to testing and quality assurance. The mention of "evaluate" alongside "experiment with" and "adopt" suggests a formal evaluation process when considering new models. The maintenance of tailored prompts for both primary and backup models indicates they've tested performance across model families. The ability to configure preferences on a per-company basis suggests they measure and optimize performance for different client needs. However, the blog post lacks specifics about evaluation methodology. How do they measure whether a backup model provides acceptable quality when failover occurs? What metrics determine whether a new model should be adopted? How do they test the failover mechanisms themselves without disrupting production traffic? These are critical LLMOps questions that the case study doesn't address, making it difficult to fully assess their evaluation maturity. ## Monitoring and Observability The case study reveals several monitoring capabilities through references to specific incidents and responses. They have latency-based alerts that triggered during the distribution-shift incident. They track provider-level errors to implement failover logic. They monitor rate limits and cache unavailability states. The ability to identify when p75+ latency jumped to over 10 seconds suggests granular latency monitoring with percentile tracking. What's less clear is how they monitor the business impact of their failover strategies. Do they track what percentage of requests use backup providers versus primary providers? How often does model-level failover occur, and does it impact customer satisfaction? Is there visibility into which components of the agent are most sensitive to provider failures? These observability questions are crucial for operating complex multi-provider systems but aren't addressed in the blog post. ## Conclusion and Assessment Gradient Labs presents a sophisticated approach to building resilient agentic systems in a high-stakes domain. Their multi-provider architecture with intelligent failover demonstrates significant LLMOps maturity and addresses real production challenges around LLM reliability. The use of Temporal for durable execution represents a thoughtful architectural choice for long-running workflows, and their multi-level failover strategy with tailored prompts shows substantial engineering investment. However, readers should approach this case study with appropriate context. This is a blog post from the company building the system, naturally emphasizing their technical achievements and sophisticated solutions. The complexity described requires substantial engineering resources and operational expertise that may not be available to all organizations. The tradeoffs around operational overhead, development velocity, and cost aren't fully explored. Several critical production concerns—cost management, quality impact of failovers, and detailed evaluation methodology—receive limited or no attention. For organizations operating in high-reliability domains with complex agentic workflows, the Gradient Labs approach offers valuable patterns worth considering. For teams in less critical applications or with smaller engineering teams, simpler approaches may provide better cost-benefit tradeoffs. The case study's most valuable contribution may be its honest discussion of ongoing challenges like distribution-shift detection, demonstrating that even sophisticated LLMOps implementations involve continuous learning and evolution rather than complete solutions.