## Overview and Context Databricks developed an AI-powered agentic platform to address database debugging challenges at massive scale. The company operates thousands of MySQL OLTP instances across hundreds of regions spanning AWS, Azure, and GCP, requiring engineers to navigate fragmented tooling including Grafana metrics, proprietary dashboards, CLI commands for InnoDB status inspection, and cloud console logs. This case study provides insights into how Databricks transitioned from manual debugging workflows to an intelligent agent-based system that has achieved company-wide adoption and demonstrated measurable impact on operational efficiency. The journey from hackathon prototype to production platform reveals important lessons about building LLM-based operational tools: the criticality of unified data foundations, the importance of rapid iteration frameworks, and the value of deeply understanding user workflows before building AI solutions. While the blog post naturally emphasizes successes, it also candidly discusses initial failures and the iterative refinement process required to achieve reliable agent behavior. ## The Problem Space Before implementing AI assistance, database debugging at Databricks suffered from three primary challenges that the organization identified through direct user research including shadowing on-call engineers: **Fragmented tooling landscape:** Engineers needed to context-switch between multiple disconnected systems during incident investigations. A typical workflow involved checking Grafana for infrastructure metrics, switching to internal Databricks dashboards for client workload patterns, executing MySQL CLI commands to inspect InnoDB internal state including transaction history and deadlock details, and finally accessing cloud provider consoles to download and analyze slow query logs. While each individual tool functioned adequately, the lack of integration created inefficient workflows that consumed significant time during critical incidents. **Context gathering overhead:** The majority of investigation time was spent determining what had changed in the environment, establishing baseline "normal" behavior, and identifying which team members possessed relevant expertise—rather than actually mitigating the incident. This information gathering phase represented significant toil that didn't directly contribute to problem resolution. **Unclear mitigation guidance:** During active incidents, engineers frequently lacked confidence about which remediation actions were safe and effective. Without clear runbooks or automation support, they defaulted to either lengthy manual investigations or waiting for domain experts to become available, both of which consumed valuable SLO budget. The organization noted that postmortems rarely surfaced these workflow gaps effectively. Teams had abundant data and tooling, but lacked the intelligent layer needed to interpret signals and guide engineers toward safe, effective actions. This realization—that the problem wasn't insufficient data but rather insufficient intelligence applied to that data—became foundational to their solution approach. ## Initial Approaches and Evolution The platform development began pragmatically with a two-day hackathon project rather than a large multi-quarter initiative. The initial prototype simply unified core database metrics and dashboards into a single view, immediately demonstrating value for basic investigation workflows despite lacking polish. This established a guiding principle of moving fast while maintaining customer obsession. Before expanding the prototype, the team conducted structured user research including interviewing service teams and shadowing on-call sessions. This research revealed that junior engineers didn't know where to begin investigations, while senior engineers found the fragmented tooling cumbersome despite their expertise. The solution evolved through several iterations, each informed by user feedback: **Version 1: Static agentic workflow** - The first production attempt implemented a static workflow following a standard debugging SOP (standard operating procedure). This approach proved ineffective because engineers wanted diagnostic reports with immediate insights rather than manual checklists to follow. **Version 2: Anomaly detection** - The team pivoted to focus on obtaining the right data and layering intelligence through anomaly detection. While this successfully surfaced relevant anomalies, it still failed to provide clear next steps for remediation. **Version 3: Interactive chat assistant** - The breakthrough came with implementing a chat interface that codifies debugging knowledge, answers follow-up questions, and transforms investigations into interactive processes. This fundamentally changed how engineers debug incidents end-to-end. This evolution demonstrates an important LLMOps lesson: the most sophisticated AI architecture won't succeed without deeply understanding user workflows and iterating based on actual usage patterns. ## Foundational Architecture A critical insight the team reached was that their ecosystem wasn't initially structured for AI reasoning across their operational landscape. Operating thousands of database instances across hundreds of regions, eight regulatory domains, and three cloud providers created specific challenges that required architectural solutions before effective AI integration became possible. **Central-first sharded architecture:** The platform implements a global Storex instance that coordinates regional shards. This provides engineers and AI agents with a single unified interface while keeping sensitive data localized within appropriate regulatory boundaries. This architecture abstracts away cloud and region-specific logic that would otherwise need to be handled explicitly in agent reasoning. **Fine-grained access control:** Authorization and policy enforcement operates at team, resource, and RPC levels, ensuring both human engineers and AI agents operate safely within appropriate permission boundaries. This centralized access control was essential for making the agent both useful and secure—without it, the system would become either too restrictive to provide value or too permissive to be safely deployed. **Unified orchestration:** The platform integrates with existing infrastructure services, providing consistent abstractions across different cloud providers and regions. This abstraction layer removes a significant reasoning burden from the AI agent while enabling humans to work with a simplified mental model. The team emphasizes that without this solid foundational architecture addressing context fragmentation, governance boundaries, and providing consistent abstractions, AI development would have encountered unavoidable roadblocks including slow iteration loops and inconsistent behavior across different deployment contexts. ## Agent Implementation and LLMOps Framework With the unified foundation established, Databricks implemented agent capabilities for retrieving database schemas, metrics, and slow query logs. The initial implementation came together quickly—within weeks they had an agent that could aggregate basic information, reason about it, and present insights to users. The significant challenge shifted to making the agent **reliable** given the non-deterministic nature of LLMs. The team needed to understand how the agent would respond to available tools, data, and prompts through extensive experimentation to determine which tools proved effective and what context should be included or excluded from prompts. **Rapid iteration framework:** To enable fast experimentation, Databricks built a lightweight framework inspired by MLflow's prompt optimization technologies, leveraging DsPy. This framework crucially decouples prompting from tool implementation. Engineers define tools as standard Scala classes and function signatures with short docstring descriptions. The LLM infers the tool's input format, output structure, and result interpretation from these descriptions. This decoupling enables rapid iteration—teams can modify prompts or swap tools without changing the underlying infrastructure handling parsing, LLM connections, or conversation state management. **Validation and evaluation:** To prevent regressions while iterating, the team created a validation framework that captures snapshots of production state and replays them through the agent. A separate "judge" LLM scores responses for accuracy and helpfulness as engineers modify prompts and tools. This automated evaluation approach addresses a core LLMOps challenge: how to systematically improve agent behavior without manual review of every change. **Agent specialization:** The rapid iteration framework enables spinning up specialized agents for different domains—one focused on system and database issues, another on client-side traffic patterns, and so on. This decomposition allows each agent to develop deep expertise in its area while collaborating with others to deliver comprehensive root cause analysis. The architecture also creates a foundation for extending AI agents to other infrastructure domains beyond databases. **Tool calling and reasoning loop:** The agent operates through an iterative loop where it decides what tools to call based on conversation context, executes those tools, and interprets results to generate responses or determine additional investigation steps. With both expert knowledge and operational context codified into its reasoning, the agent can extract meaningful insights and actively guide engineers through investigations. ## Production Capabilities and Impact The deployed system delivers several concrete capabilities that demonstrate the maturity of the LLMOps implementation. Within minutes, the agent surfaces relevant logs and metrics that engineers might not have considered examining independently. It connects symptoms across different system layers—for example, identifying which workspace is driving unexpected load and correlating IOPS spikes with recent schema migrations. Critically, the agent explains underlying cause and effect relationships and recommends specific next steps for mitigation rather than simply presenting data. The measured impact has been substantial according to the organization's metrics. Individual investigation steps that previously required switching between dashboards, CLIs, and SOPs can now be answered through the chat assistant, cutting time spent by up to 90% in some cases. The learning curve for new engineers has dropped sharply—new hires with zero infrastructure context can now jump-start database investigations in under 5 minutes, something described as "nearly impossible" with the previous tooling. The platform has achieved company-wide adoption with strong qualitative feedback from engineers. One staff engineer noted that the assistant "saves me a ton of time so that I don't need to remember where all my queries dashboards are," while another described it as "a step change in developer experience" and noted they "can't believe we used to live in its absence." From an architectural perspective, the platform establishes a foundation for the next evolution toward AI-assisted production operations. With data, context, and guardrails unified, the organization can now explore how agents might assist with restores, production queries, and configuration updates—moving beyond investigation toward active operational intervention. ## Critical Assessment and Balanced Perspective While the case study presents an impressive success story, several aspects warrant balanced consideration: **Validation methodology:** The use of a "judge" LLM to score agent responses addresses a real challenge in LLMOps, but this approach has known limitations. LLMs evaluating other LLMs can exhibit biases and may not always align with human judgment, particularly for complex technical assessments. The case study would be strengthened by discussion of human evaluation samples or metrics showing correlation between judge LLM scores and actual engineer satisfaction or incident resolution effectiveness. **Quantitative claims:** The "up to 90%" time reduction claim is notable but presented without methodological details. This appears to represent best-case scenarios for specific tasks rather than average improvements across all debugging activities. More complete reporting would include median improvements, variance across different issue types, and clearer specification of what baseline and comparison conditions were used. **Generalization limits:** The solution is deeply tailored to Databricks' specific infrastructure context—MySQL databases with particular tooling integrations and operational patterns. While the architectural principles may transfer, organizations with different technology stacks would need substantial adaptation rather than direct application of this approach. **Production reliability:** The case study doesn't address failure modes, fallback mechanisms, or how the system handles situations where the agent provides incorrect guidance. For production operational systems, understanding error cases and mitigation strategies is as important as understanding successful cases. **Cost considerations:** Operating LLM-based agents at scale for company-wide debugging workflows likely involves non-trivial inference costs. The absence of cost discussion may indicate these costs are manageable relative to engineering time savings, but this represents an important consideration for organizations evaluating similar approaches. Despite these limitations in the presentation, the case study demonstrates genuine LLMOps maturity through its focus on foundational architecture, systematic iteration frameworks, and concrete operational deployment rather than just proof-of-concept demonstrations. ## Key Takeaways and Lessons The Databricks team distills their experience into three core lessons that reflect genuine LLMOps insights: **Rapid iteration is essential:** Agent development improves through fast experimentation, validation, and refinement. The DsPy-inspired framework enabling quick evolution of prompts and tools without infrastructure changes proved critical to achieving reliable agent behavior. **Foundation determines iteration speed:** Unified data, consistent abstractions, and fine-grained access control removed the biggest bottlenecks to agent development. The quality of the underlying platform determined how quickly the team could iterate on AI capabilities. **Speed requires correct direction:** The team emphasizes they didn't set out to build an agent platform initially. Each iteration followed user feedback and incrementally moved toward the solution engineers actually needed. This reflects mature product thinking where technical capability serves clearly understood user needs rather than being pursued for its own sake. The broader insight the team offers is that building internal platforms requires treating internal customers with the same rigor as external ones—through customer obsession, simplification through abstractions, and elevation through intelligence. This approach bridges the gap between product and platform teams that often operate under very different constraints within the same organization. ## Technical Architecture Insights Several architectural decisions reflect LLMOps best practices worth highlighting: The decoupling of tool definitions from prompt engineering through the DsPy-inspired framework represents a sophisticated understanding of agent development workflows. By allowing tools to be defined as normal code with docstrings rather than requiring prompt engineering for each tool integration, the system dramatically reduces the friction of expanding agent capabilities. The use of specialized agents for different domains rather than a single monolithic agent demonstrates understanding of how to manage complexity in agentic systems. This architectural pattern allows for deeper expertise in specific areas while maintaining the ability to collaborate across agents for complex investigations requiring multiple perspectives. The central-first sharded architecture with regional data locality shows how to build AI systems that respect regulatory and data governance requirements while still providing unified interfaces. This represents a practical solution to a common challenge in global-scale systems where naive centralization would violate data residency requirements. The validation framework using production state snapshots and judge LLMs creates a systematic approach to regression testing for non-deterministic systems—one of the fundamental challenges in LLMOps. While this approach has limitations as noted above, it represents a pragmatic solution enabling continuous improvement without purely manual evaluation. ## Conclusion This case study presents a substantive example of LLMOps at scale within a complex operational context. Databricks successfully transitioned from fragmented manual workflows to an intelligent agent-based system that has achieved measurable impact on debugging efficiency and engineer onboarding. The journey from hackathon prototype to production platform reveals important lessons about the relationship between foundational architecture and AI capability, the criticality of rapid iteration frameworks, and the importance of deeply understanding user workflows. While the presentation emphasizes successes and could benefit from more detailed discussion of limitations, failure modes, and costs, the technical approach demonstrates genuine maturity in areas like tool abstraction, agent specialization, automated evaluation, and production deployment. The case provides valuable insights for organizations considering similar agent-based approaches to operational workflows, particularly around the foundational work required before AI integration becomes tractable and the importance of iterative refinement based on actual usage patterns.