NVIDIA's Applied AI Lab for DGX Cloud developed LLo11yPop, a hierarchical agent system for managing large-scale GPU infrastructure. The problem involved monitoring and optimizing hundreds of GPU clusters with complex failure modes, resource allocation constraints, and the need for proactive incident detection. The solution employed a multi-tier agent architecture with specialized worker agents for data retrieval, analyst agents for reasoning, orchestrator agents for coordination, and tool agents for actions. By constraining individual agents to specific tasks and using deterministic fallbacks where needed, the system achieved production-grade reliability for GPU fleet management. The architecture demonstrated that balancing agentic discovery with deterministic tools, coupled with comprehensive evaluation strategies and rare context integration, enables scalable AI operations in high-stakes environments.
Aaron Erickson from NVIDIA’s Applied AI Lab for DGX Cloud presented a comprehensive case study on designing and deploying production-grade AI agent systems at scale. The presentation, delivered at QCon San Francisco 2026, chronicles the evolution from early experiments with generative AI (including an organizational restructuring tool built in 2023) to the development of LLo11yPop, a sophisticated multi-agent system for GPU infrastructure management. The case study provides valuable insights into the practical challenges of moving from AI prototypes to reliable production systems, emphasizing the critical balance between agentic discovery and deterministic certainty.
Erickson’s journey began before joining NVIDIA, when he worked on a startup building an AI-powered organizational design system. In April 2023, following the release of GPT-4, they built a ChatGPT plugin that could analyze organizational structures and propose reorganizations. This early experiment demonstrated both the promise and peril of agentic AI—while the system could generate restructuring plans and draft communication emails, it highlighted the need for constraints, human oversight, and domain-specific knowledge. This experience laid the foundation for understanding how to build reliable AI systems for high-stakes production environments.
Upon joining NVIDIA in 2023, Erickson found himself working on systems to manage GPU allocation across hundreds of research teams. The problem space proved remarkably similar to HR systems: allocating scarce resources (GPUs instead of headcount), managing complex hierarchies (cloud providers, regions, and blocks instead of organizational structures), tracking utilization (GPU jobs instead of employees), and ensuring optimal performance through comprehensive observability.
LLo11yPop (Large Language Model for Observability) emerged as NVIDIA’s solution for managing GPU infrastructure at scale. The system’s core insight was that reliability comes not from building one monolithic AI that tries to do everything, but from constructing a hierarchy of purpose-built agents, each specialized for specific tasks with well-defined boundaries.
The architecture consisted of five primary layers:
Worker Agents formed the foundation, handling low-level data retrieval tasks. These agents were highly constrained, designed to convert natural language queries into specific database operations (SQL, Elasticsearch queries, or API calls). The key innovation was deliberate constraint—each worker agent knew how to interact with exactly one data source or perform one specific query pattern. By using flat database schemas instead of complex normalized structures with joins, NVIDIA found they could achieve much higher reliability. Worker agents essentially performed “low-level intellectual horsepower applied at scale,” examining thousands of GPU clusters for specific conditions like wattage fluctuations or network connectivity issues. The system employed basic RAG (Retrieval-Augmented Generation) with carefully curated examples to ensure consistent output formatting.
Analyst Agents operated at a higher level, aggregating insights from multiple worker agents. These agents didn’t interact directly with databases; instead, they understood domain-specific concepts like thermal failure patterns in H100 GPUs or performance degradation indicators. Analyst agents could correlate data from multiple sources, identifying patterns that might indicate emerging issues. They were essentially specialized in reasoning about specific problem domains—one analyst agent might focus on heat management, another on network performance, another on power distribution. This specialization allowed each agent to develop deep expertise in its area while maintaining manageable context windows and high accuracy.
Orchestrator Agents coordinated the work of multiple analyst agents to achieve broader goals. Given an objective like “optimize fleet utilization” or “identify potential failures before they occur,” orchestrator agents would determine which analyst agents to consult, gather their recommendations, and synthesize insights across different perspectives. This tier implemented what Erickson called “goal-oriented behavior”—taking a high-level objective and breaking it down into specific tasks that could be delegated to specialized agents.
Tool Agents provided the action layer, translating recommendations into concrete operations. However, critically, these agents didn’t automatically execute changes. Instead, they generated recommendations that were surfaced to human operators through systems like Slack or Jira. A tool agent might know how to create a properly formatted Jira ticket for GPU maintenance or draft a Slack notification about an anomaly, but the actual execution remained under human control. This design reflected a core principle: in production systems dealing with expensive resources, human oversight is non-negotiable.
Director Agents represented the aspirational top tier—agents capable of observing, orienting, deciding, and acting in a closed OODA loop. While Erickson acknowledged that few organizations are running fully autonomous director agents in production, he positioned this as the evolution path for mature systems. The key to making director agents viable would be constraining their operational domain, building comprehensive evaluation systems, and gradually increasing autonomy as confidence in the system grows.
One of the most valuable insights from the LLo11yPop deployment concerned what Erickson termed “rare context”—company-specific terminology, domain knowledge, and operational patterns that aren’t present in general LLM training data. When operations teams asked questions like “where are the zombie nodes?”, the system initially struggled because “zombie” wasn’t a standard technical term with a universal definition. It might mean nodes with no network connectivity, nodes with failed cooling, nodes that accept work but don’t complete it, or something else entirely depending on the organization’s operational culture.
NVIDIA addressed this through deliberate context engineering. They built domain-specific fine-tuning on top of base models, incorporated examples of company terminology into prompts, and created specialized agent personas that understood NVIDIA’s specific operational language. This investment in rare context proved essential for moving from 80% accuracy (acceptable for demos) to the 95%+ accuracy needed for production operations.
The lesson extends beyond just terminology. Rare context includes understanding which metrics matter most for specific workloads, knowing the historical incident patterns that predict future failures, and recognizing the operational constraints that make certain solutions preferable to others. These elements can’t be effectively captured through zero-shot prompting or general-purpose models—they require intentional knowledge engineering and continuous refinement based on operational feedback.
NVIDIA implemented a comprehensive evaluation framework modeled on the traditional software testing pyramid, adapted for LLM systems. This multi-layered approach proved critical for maintaining reliability as the system scaled.
Unit-Level Evaluations focused on individual worker agents, testing whether they correctly translated natural language queries into accurate database operations and returned properly formatted results. These evaluations ran frequently, were computationally inexpensive, and could quickly catch regressions when models or prompts were updated. NVIDIA employed LLM-as-a-judge techniques for these evaluations, recognizing that exact string matching wouldn’t work (since LLMs might phrase “George Washington” in multiple valid ways) but semantic equivalence could be reliably assessed by another LLM.
Integration-Level Evaluations tested analyst agents working with multiple worker agents. These evaluations verified that agents could correctly aggregate information from diverse sources and reason appropriately about multi-faceted problems. They ran less frequently than unit tests due to higher computational costs but were essential for catching issues in how agents composed and synthesized information.
End-to-End Evaluations tested entire workflows from initial query through to recommended actions. These expensive evaluations ran on representative scenarios, comparing system outputs against known ground truth for complex real-world situations. NVIDIA sampled production scenarios, created evaluation datasets from historical incidents, and continuously measured whether the system would have correctly identified and recommended appropriate responses.
The test pyramid approach allowed NVIDIA to maintain rapid iteration velocity while ensuring reliability. When changes were made to prompts, model versions, or agent configurations, the multi-tier evaluation system could quickly identify where problems emerged and at what level of the stack they occurred.
A recurring theme in Erickson’s presentation was the concept of “determinism as an off-ramp from the road to hell”—the idea that while agentic systems enable valuable discovery and reasoning, production reliability often requires constraining or replacing agentic behavior with deterministic rules once patterns are understood.
When NVIDIA’s agents repeatedly struggled with certain query types—perhaps complex joins that the LLM couldn’t reliably formulate, or counting operations that produced inconsistent results—they would create deterministic fallback rules. These rules might say “when asked about GPU counts in a specific context, use this parameterized query template” or “for financial calculations, always delegate to Python code rather than attempting to reason about numbers.” This approach drew inspiration from software engineering practices like Cursor rules, where developers provide explicit guidance about how to handle specific scenarios.
The philosophy wasn’t “replace AI with traditional programming” but rather “use AI for discovery and reasoning, use deterministic systems for reliability where patterns are well-understood.” NVIDIA found that this hybrid approach delivered the best of both worlds: the ability to handle novel situations and discover new patterns, combined with the reliability needed for production operations.
Through their work, NVIDIA identified several recurring agent patterns that proved valuable across different problem domains:
Worker Agents excel at “painting rocks on the beach”—applying modest intellectual effort at scale across large datasets. Tasks like examining 100,000 GPU clusters for specific anomalies, scanning documents for particular patterns, or checking configurations across a fleet are ideal worker agent applications. They’re characterized by low per-item complexity but high volume requirements.
Ruminative Agents perform extended reasoning over exceptions or anomalies identified by worker agents. These agents might run overnight, exploring multiple perspectives on a problem, conducting simulated experiments, or identifying subtle correlations that wouldn’t be obvious from simple analysis. The key characteristic is using extended inference time (and computational budget) to discover non-obvious insights.
Middle Manager Agents take high-level goals (“reduce latency in customer-facing services”) and decompose them into specific actionable recommendations by coordinating multiple specialized agents. They aggregate perspectives, prioritize actions based on constraints, and package recommendations for human decision-makers or execution systems.
Consultant Agents observe patterns across the system itself—meta-agents that watch how other agents perform and recommend optimizations to the agent architecture itself. They might identify opportunities to replace frequently-repeated agentic reasoning with deterministic rules, suggest new agents that should be created for common patterns, or flag areas where agents consistently struggle.
Tool Selector Agents handle routing and orchestration, determining which specialized tools or agents are appropriate for a given task. These implement the pattern now seen in systems like Claude’s MCP (Model Context Protocol) or ChatGPT’s plugin system—taking a high-level request and determining the right sequence of tool invocations to fulfill it.
Director Agents represent the aspirational autonomous tier—agents that can observe systems, orient to problems, decide on actions, and execute changes in a closed loop. While Erickson acknowledged these aren’t widely deployed in production yet, he positioned them as the natural evolution for mature agent systems operating in constrained domains with comprehensive safety measures.
Several practical insights emerged from NVIDIA’s production deployment:
Simplified Schemas: LLMs performed significantly better when querying wide, flat database tables rather than normalized schemas requiring complex joins. This mirrors the observation that early-career engineers often benefit from starting with simpler data structures before tackling complex relational designs.
Explicit Context Engineering: Rather than relying on general knowledge, NVIDIA invested heavily in providing agents with specific examples, company terminology definitions, and domain-specific reasoning patterns. This “rare context” proved essential for achieving production-grade accuracy.
Guardrails Through Identity: Rather than running agents with the same permissions as users, NVIDIA created specific identities for agents with narrowly scoped privileges. An agent might have read-only access to specific database tables and the ability to create Jira tickets in certain projects, but nothing more. This principle of least privilege proved essential for safe operations.
Human-in-the-Loop at Transaction Boundaries: Any operation with significant consequences—spending money, moving GPU allocations, changing configurations—required explicit human approval. The agents could recommend, draft, and prepare actions, but couldn’t execute them autonomously.
Grounding Frequency as a Dial: In reasoning-intensive workflows (similar to deep research systems), NVIDIA found that the frequency of grounding—how often the system checked back against source data versus pure reasoning—was a critical tuning parameter. Too much grounding made analysis slow and mechanical; too little led to hallucinations and reasoning drift.
Operational Metrics Matter: Beyond traditional accuracy metrics, NVIDIA tracked how often agents were consulted, which recommendations operators accepted or rejected, and where agents consistently struggled. These operational signals drove continuous improvement.
While the initial LLo11yPop system was built before modern agent frameworks were widely available, NVIDIA leveraged basic prompt engineering, RAG techniques, and custom orchestration. Erickson noted that what initially took their team five months to build could later be replicated in approximately six hours using modern frameworks like LangChain and MCP servers—a testament to how rapidly the tooling landscape evolved. However, the early development work provided invaluable insights into what makes agent systems reliable, insights that informed both NVIDIA’s subsequent work and their contributions to the broader AI community.
The system used NVIDIA’s own NIM (NVIDIA Inference Microservice) infrastructure for model deployment, though Erickson emphasized the architecture was model-agnostic. NVIDIA eventually developed Nemotron models specifically optimized for reasoning tasks based on learnings from LLo11yPop and similar systems.
NVIDIA productized many learnings from LLo11yPop into what they call “blueprints”—reference architectures that organizations can adapt for their own use cases. One example is AIQ, a portable deep research assistant that implements the grounded reasoning patterns discovered in LLo11yPop. Unlike commercial deep research products that operate over public internet data, AIQ is designed to run within organizations, with access to proprietary databases, documentation repositories, and internal knowledge bases.
This approach reflects NVIDIA’s broader strategy: develop novel techniques internally for their own operations, validate them at scale, then share patterns openly so the community can build on them. The company benefits from ecosystem innovation while establishing best practices for AI infrastructure.
Erickson’s presentation culminated in a coherent philosophy about AI reliability: the most reliable systems will be those that thoughtfully combine agentic discovery with deterministic certainty. Agents excel at handling novel situations, discovering patterns humans haven’t encoded, and reasoning through complex trade-offs. Deterministic systems excel at repeatedly executing well-understood operations with guaranteed behavior.
The art of production AI engineering lies in knowing which problems belong in which category, building the infrastructure to support both modes, and creating clear interfaces between them. Worker agents might discover through reasoning that certain thermal patterns predict GPU failures; once validated, that pattern becomes a deterministic rule checked by monitoring systems. Analyst agents might explore different optimization strategies for resource allocation; once a strategy proves effective, it becomes a standard operating procedure.
This philosophy pushes back against two extremes visible in the industry: AI maximalism (everything should be solved by throwing more tokens at bigger models) and AI skepticism (these systems can’t be trusted in production). Instead, it offers a pragmatic middle path: use AI where its strengths—flexibility, discovery, reasoning over uncertainty—provide value, and constrain or replace it with deterministic approaches where reliability requirements exceed what current AI can deliver.
The case study demonstrates several critical requirements for production LLMOps:
Comprehensive evaluation frameworks that test at multiple levels of system hierarchy are non-negotiable. Unit tests catch basic failures, integration tests catch composition problems, and end-to-end tests validate real-world performance.
Human feedback loops that capture operator acceptance, rejection rationale, and edge cases feed continuous improvement. The system should get better through use, but only if usage patterns are instrumented and analyzed.
Graceful degradation paths ensure that when agents fail or produce uncertain results, the system can fall back to deterministic approaches or escalate to human operators rather than failing silently.
Domain specialization through rare context engineering provides the bridge between general-purpose models and production effectiveness in specific organizational contexts.
Identity and privilege management adapted from traditional security practices remain essential—agents should operate with minimal necessary permissions, not inherit broad user privileges.
Looking forward, Erickson suggested the industry is moving toward more autonomous director agents operating in constrained domains, more sophisticated evaluation techniques beyond basic LLM-as-a-judge approaches, and better frameworks for managing the hybrid deterministic-agentic systems that production reliability demands. The Waymo self-driving car served as his example of what’s possible when agent systems are built with appropriate constraints, comprehensive testing, and years of refinement—demonstrating that truly autonomous AI agents do work in production when properly engineered.
Notion, a knowledge work platform serving enterprise customers, spent multiple years (2022-2026) iterating through four to five complete rebuilds of their agent infrastructure before shipping Custom Agents to production. The core problem was enabling users to automate complex workflows across their workspaces while maintaining enterprise-grade reliability, security, and cost efficiency. Their solution involved building a sophisticated agent harness with progressive tool disclosure, SQL-like database abstractions, markdown-based interfaces optimized for LLM consumption, and a comprehensive evaluation framework. The result was a production system handling over 100 tools, serving majority-agent traffic for search, and enabling workflows like automated bug triaging, email processing, and meeting notes capture that fundamentally changed how their company and customers operate.
Digits, a company providing automated accounting services for startups and small businesses, implemented production-scale LLM agents to handle complex workflows including vendor hydration, client onboarding, and natural language queries about financial books. The company evolved from a simple 200-line agent implementation to a sophisticated production system incorporating LLM proxies, memory services, guardrails, observability tooling (Phoenix from Arize), and API-based tool integration using Kotlin and Golang backends. Their agents achieve a 96% acceptance rate on classification tasks with only 3% requiring human review, handling approximately 90% of requests asynchronously and 10% synchronously through a chat interface.
This panel discussion features three AI-native companies—Delphi (personal AI profiles), Seam AI (sales/marketing automation agents), and APIsec (API security testing)—discussing their journeys building production LLM systems over three years. The companies address infrastructure evolution from single-shot prompting to fully agentic systems, the shift toward serverless and scalable architectures, managing costs at scale (including burning through a trillion OpenAI tokens), balancing deterministic workflows with model autonomy, and measuring ROI through outcome-based metrics rather than traditional productivity gains. Key technical themes include moving away from opinionated architectures to let models reason autonomously, implementing state machines for high-confidence decisions, using tools like Pydantic AI and Logfire for instrumentation, and leveraging Pinecone for vector search at scale.
