## Overview This case study captures insights from a presentation by Craig, who leads product management for Databricks and previously held leadership positions at Google (founding Vertex AI) and AWS (founding SageMaker). The presentation focuses on how Databricks enables enterprise customers to deploy LLMs into production environments with proper governance, evaluation, and integration with complex data estates. The talk emphasizes the gap between prototyping AI capabilities and deploying them into critical production paths within large organizations, particularly in highly regulated industries like finance and healthcare. Databricks positions itself as a multi-cloud data platform serving tens of thousands of customers, created by the teams behind open-source projects including Spark, MLflow, and Delta. The company's approach to LLMOps centers on what they call "Mosaic AI," which provides capabilities specifically designed for production deployments where financial and reputational risks are significant. ## Core Problem Statement The presenter articulates a fundamental challenge facing large enterprises: these organizations typically have nightmarish data scenarios resulting from years of acquisitions, with data scattered across multiple clouds, vendors, and services. This fragmentation creates multiple challenges for LLM deployment. Organizations often have numerous data warehouses with siloed expertise, where specialists in one system don't understand others. Additionally, streaming engineers aren't necessarily GenAI experts, creating knowledge gaps that prevent effective data integration with LLM systems. The talk emphasizes a critical distinction between "general intelligence" (foundational LLM capabilities) and "data intelligence" (connecting LLMs to enterprise data estates). While both are valuable, enterprises seeking to automate systems or drive deeper insights inevitably need to connect LLMs to their proprietary data. The challenge is particularly acute because many organizations are attempting to build deterministic systems using highly probabilistic components (LLMs), creating reliability concerns for high-stakes use cases. ## Primary Case Study: FactSet Query Language Translation FactSet, a financial services company that sells financial data about companies to banks and hedge funds, provides the most detailed technical case study in the presentation. FactSet developed their own proprietary query language (FQL - FactSet Query Language), which customers had to learn to access their data. This created a significant barrier to entry and customer experience issue. ### Initial Implementation and Results When generative AI became prominent, FactSet recognized an opportunity to translate natural language English into FQL, potentially eliminating the learning curve for customers. Their initial approach followed what the presenter somewhat dismissively calls the "one-click RAG button" methodology. They constructed a massive prompt containing numerous examples and documentation, paired with a large vector database also populated with examples and documentation. This initial implementation delivered disappointing results: 59% accuracy with approximately 15 seconds of latency. The presenter emphasizes the latency metric not just as a user experience concern but as a proxy for cost, since in the GenAI world, organizations are essentially paying for compute time. The 59% accuracy was characterized as "slightly better than a coin flip," making the system unsuitable for production deployment despite the excitement around having a GenAI solution. ### Databricks Intervention and Optimization Working with Databricks, FactSet took a fundamentally different architectural approach. Rather than relying on a single massive prompt, they decomposed the task into individual components, creating what the presenter describes as "something of an agent of a multi-node multi-step chain or process." This decomposition was strategic: it enabled performance tuning at each discrete step of the problem-solving process. The results were dramatic: accuracy improved from 59% to 85%, while latency was reduced from 15 seconds to 6 seconds. This represents both a significant quality improvement and a substantial cost reduction (60% reduction in latency/cost). At 85% accuracy, FactSet felt comfortable deploying the system to existing customers. Moreover, at this point FactSet indicated they understood the methodology well enough to continue optimization independently. The presenter notes that in subsequent conversations, FactSet had achieved accuracy "into the 90s" and was planning to transition to Claude as their next roadmap item. ### Architectural Implications The FactSet case exemplifies a key finding from Berkeley's Artificial Intelligence Research lab, which the presenter references. Berkeley researchers examined popular AI systems in production and found that none relied on simple single-input-to-single-output architectures. Instead, all successful production systems employed complex multi-node, multi-step chains to achieve reliable outcomes. This finding validates Databricks' approach of encouraging customers toward composable agentic architectures rather than monolithic prompt-based solutions. ## Databricks LLMOps Platform Architecture The presentation outlines Databricks' comprehensive approach to production LLM deployment, built on two foundational pillars: governance and evaluation. The platform integrates Claude natively across all major clouds (Azure, AWS, GCP), enabling customers to build sophisticated agents while maintaining enterprise-grade controls. ### Governance Framework Databricks treats AI agents as principles (entities with permissions) within the data stack, enabling granular access control. The governance model extends across multiple dimensions: - **Data Access Governance**: Fine-grained control over what data the LLM can access, critical given the fragmented data estates typical in large enterprises - **Model Access Governance**: Control over which models can be used in different contexts - **Tool Access Governance**: Governance over what functions and tools the agent can invoke - **Query Governance**: Control over the types of queries that can be executed The presenter notes that MCP (Model Context Protocol) servers are not yet governed in their system but hints at upcoming announcements at an imminent conference. This governance framework addresses a key concern raised by regulated industries: multiple banks in a meeting the presenter attended were prototyping with Claude, but one bank representative indicated they couldn't use generative AI due to lack of controls. The difference wasn't the technology itself but rather whether organizations had implemented sufficient governance to safely deploy the technology. ### Tool Calling and Deterministic Design Tool calling emerges as a critical capability for building production-grade systems. The presenter describes a common pattern where LLMs serve as classifiers choosing between six to eight different execution paths or tools. These tools might be other agents, SQL queries, or any parameterizable function. This creates a decision tree architecture that reduces entropy and increases determinism. However, the presenter candidly admits that before the Databricks-Anthropic partnership, tool calling reliability was insufficient. Models would fail to select obviously correct tools, undermining the deterministic design goals. Claude's improved tool calling capabilities fundamentally changed this dynamic, making it feasible to build "quasi deterministic systems using a highly probabilistic backend." The presenter characterizes Claude as "completing the puzzle" by providing frontier LLM capabilities with the reliability needed for production deployment. ### Data Integration Layer The platform connects LLMs to enterprise data through vector stores and feature stores. This integration is positioned as the key differentiator from hyperscaler offerings. The presenter, drawing on experience at both AWS and GCP, argues that while MLOps took an order of magnitude off development time, the next order of magnitude reduction comes from deeply integrating AI and data layers—something hyperscalers haven't achieved to the same degree. ### Evaluation Framework Databricks' evaluation platform addresses a fundamental gap in LLM deployment: measuring system quality. The presenter expresses frustration with customers claiming their systems are "pretty good" without quantification. In contrast, a global logistics provider that reported 85% accuracy was celebrated as an exception demonstrating proper engineering discipline. The evaluation system operates on several principles: - **Golden Dataset Testing**: Customers provide reference datasets against which system performance is measured - **LLM Judges**: Databricks provides proprietary LLM judges to assess response quality across multiple dimensions, including a "safe score" metric (clarified in Q&A as more of a guardrail measure than adversarial testing) - **Subject Matter Expert Interface**: A simplified UI allows non-technical domain experts to correct responses and refine prompts, recognizing that application developers are often not subject matter experts - **Iterative Improvement**: The system enables "hill climbing" toward higher accuracy through systematic testing The presenter characterizes this as "unit testing the agent" but suggests it's more sophisticated, involving comprehensive search across the expected question space and granular performance validation. Much of this capability is available in open-source MLflow, though the LLM judges are proprietary to the Databricks platform. This hybrid approach allows organizations to use the evaluation methodology whether or not they're full Databricks customers. ## Secondary Case Study: Databricks Internal Use - Analyst Questionnaires Databricks applies its own platform internally for a use case that directly impacts the product management team: responding to analyst questionnaires from firms like Gartner and Forrester. These questionnaires are comprehensive—the last Gartner questionnaire contained 180 questions and resulted in a 450-page response document requiring hundreds of hours from product managers, engineers, and marketing staff. ### Implementation Details Databricks built an internal system using Claude that ingests their documentation, blog posts, and previous analyst questionnaire responses. When new questionnaires arrive, they process them through the system, which generates responses of surprising quality. The presenter notes that while they still review and occasionally correct outputs, the system produces something closer to a "final draft" rather than a rough draft. ### Model Evolution and Results The development process involved multiple iterations through different model options. They started with open-source models, then moved to non-Anthropic commercial models, and finally to Claude. The critical inflection point came with Claude adoption: for the first time, the system produced outputs that could ship without modification. This capability transformed analyst questionnaire response from a massive coordination effort into primarily an editing task, saving substantial time while maintaining quality. The presenter's enthusiasm for this use case is personal—it "makes my life way better"—and Databricks published a blog post about the implementation, suggesting they view it as a reference architecture others could replicate for similar document-intensive workflows. ## Additional Enterprise Use Case: Block's Goose The presentation briefly discusses Block (the payments company) and their development of Goose, an open-source agentic development environment. Goose integrates Claude with connections to Block's systems and data, creating an accelerated developer experience that goes well beyond traditional code completion tools. Block reports impressive adoption metrics: 40-50% weekly user adoption increase and 8-10 hours saved per developer per week. The presenter positions Goose as an example of purpose-built systems that leverage LLM capabilities within enterprise contexts, achieving productivity gains by deeply integrating with organizational data and workflows. The open-source nature of Goose also represents Databricks' broader strategy of building on and contributing to open-source ecosystems. ## Technical Philosophy and Design Patterns Several recurring technical themes emerge throughout the presentation: ### Composable Architecture Over Monolithic Prompts The FactSet case demonstrates the superiority of decomposed, multi-step architectures over single massive prompts. This approach trades simplicity for tunability, enabling optimization at each step. The presenter explicitly encourages companies toward "composable agentic approaches," particularly when building for high-risk environments requiring deterministic behavior. ### Entropy Reduction as Core Goal A consistent theme is "driving as much entropy out of these systems as possible" to achieve determinism. This motivates both the multi-step architecture (each step can be controlled and tuned) and the sophisticated tool calling (classifier-style routing to specific execution paths). The presenter acknowledges that while larger models like Claude 3.7 might handle more in a single pass, the concern is whether engineers would have sufficient "knobs" to correct errors beyond prompt adjustment. ### Production Risk as Design Driver The presentation repeatedly distinguishes between low-stakes prototyping ("a chatbot for you and your buddies to search over documents") and high-stakes production deployment involving "financial or reputational risk." This distinction drives all architectural decisions. The presenter is explicit that one-click RAG is fine for personal tools but inadequate for production systems where errors have serious consequences. ### Cost-Latency-Quality Tradeoffs While not extensively elaborated, the FactSet case illustrates these tradeoffs clearly. The initial system's 15-second latency represented both poor user experience and high cost. The optimized system achieved both better quality (85% vs 59% accuracy) and lower cost/latency (6 vs 15 seconds), demonstrating that proper architecture can improve multiple dimensions simultaneously rather than requiring tradeoffs. ## Industry Context and Competitive Positioning The presenter positions Databricks against multiple competitive contexts. For point solutions within specific GenAI capabilities, they acknowledge competitors like Galileo and Patronius for evaluation. However, Databricks differentiates on the integration between AI and data layers, arguing this integration is where the next order-of-magnitude productivity improvement originates. The presenter's career trajectory—founding roles at AWS SageMaker and Google Vertex AI before joining Databricks—lends credibility to claims about hyperscaler limitations. The implication is that hyperscalers have strong individual components but lack the deep AI-data integration that enterprises need for production LLM deployment. The presentation also reveals the rapid evolution of the LLMOps ecosystem. The Databricks-Anthropic partnership specifically addressed tool calling reliability issues that previously prevented production deployment of certain architectures. The upcoming MCP server governance support indicates continued platform evolution to keep pace with emerging standards and protocols. ## Regulated Industry Adoption A recurring theme involves adoption patterns in heavily regulated industries, particularly banking and healthcare. The presenter describes a meeting with 10-12 banks where more than half were prototyping with Claude. The one bank representative who said they couldn't use generative AI was "laughed at by the others," illustrating how quickly governance and control frameworks have enabled previously cautious organizations to adopt LLM technology. This shift represents a maturation of LLMOps practices. Organizations no longer need to wait for technology to "come to them"—instead, with proper governance frameworks, they can proactively adopt frontier models like Claude while maintaining regulatory compliance. The distinction between organizations deploying versus avoiding LLMs increasingly comes down to whether they've implemented appropriate controls rather than inherent technology limitations. ## Critical Assessment and Balanced Perspective While this presentation obviously serves as marketing for Databricks' platform and the Claude partnership, several elements suggest substantive technical value: The FactSet results (59% to 85% accuracy, 15s to 6s latency) are specific and verifiable claims rather than vague assertions. The Berkeley AI Research citation provides academic grounding for architectural recommendations. The presenter's admission that tool calling was insufficient before Claude, and that various model experiments preceded finding an effective solution, suggests intellectual honesty rather than pure salesmanship. However, several claims require skepticism. The characterization of hyperscaler offerings as lacking deep AI-data integration may be overstated given the extensive integration work at AWS and GCP. The evaluation framework, while sensible, isn't unique to Databricks—similar approaches exist across the LLMOps ecosystem. The MCP governance gap suggests the platform is still catching up to emerging standards. The "unit testing the agent" characterization that the presenter initially resists may actually be more accurate than they'd like to admit. While comprehensive, the evaluation approach described is fundamentally about systematic testing against expected inputs and outputs—sophisticated unit testing at scale. The presentation also doesn't deeply address several critical production concerns: monitoring and observability in production, managing model version updates, handling edge cases and failure modes, or managing costs at scale. These omissions suggest the presentation focuses on deployment architecture while leaving operational concerns less explored. ## Conclusion and Strategic Implications This case study illustrates the maturation of enterprise LLMOps from experimental prototyping to production deployment in high-stakes environments. The technical patterns—decomposed architectures, granular governance, systematic evaluation, and tight data integration—represent emerging best practices for organizations moving beyond proof-of-concept into operational systems. The emphasis on measurement and evaluation reflects a healthy evolution toward engineering discipline in LLM deployment. The presenter's enthusiasm for meeting a customer who could quantify their system's accuracy highlights how rare such rigor remains, even as it becomes essential for production deployment. The Databricks platform approach—combining open-source foundations (MLflow, Spark, Delta) with proprietary capabilities (LLM judges, governance frameworks) and partnership integrations (Claude)—represents one viable path for enterprise LLMOps. Whether it's the optimal path remains debatable, but the architectural principles and case study outcomes provide valuable reference points for organizations designing their own LLM production systems.