Intuit developed a specialized LLM-based financial advisory system to address the fundamental limitation of off-the-shelf frontier models: they provide fluent but unreliable financial advice because they lack grounding in real business outcomes. The problem manifested when these models gave contradictory or potentially harmful recommendations despite having full access to business financial data. Intuit's solution involved creating millions of business trajectories from their rich data sources (QuickBooks, TurboTax, Credit Karma, Mailchimp), extracting state-action-outcome vectors from over 100,000 businesses, and training a reinforcement learning model to understand which actions lead to successful outcomes in similar situations. They then trained a custom LLM using this grounded approach, enabling it to outperform frontier models in head-to-head testing despite being smaller and cheaper. The resulting AI business advisor is currently in beta, proactively suggesting evidence-based actions to small and medium businesses with explanations grounded in what similar businesses did and achieved.
Intuit, a major financial technology company providing products like QuickBooks, TurboTax, Credit Karma, and Mailchimp, recognized a critical problem with using off-the-shelf large language models for financial advisory: these models consistently provide fluent, confident answers about money that sound reasonable but are fundamentally unreliable and sometimes harmful. The speaker, Udy Menquez, a Principal Product Manager at Intuit leading financial intelligence and advisory systems, illustrated this through personal experience where changing minor assumptions caused frontier models to completely flip their investment recommendations from option A to option B and back again.
Through extensive research across approximately 100,000 business situations and timeframes, Intuit documented systematic failures of frontier models when providing financial advice. In one example, a landlord with negative cash flow and declining profit asked how to improve profitability. A frontier model with access to all the business’s financial data recommended acquiring a second rental property, which would be extremely risky for a business already in the red. In contrast, Intuit’s outcome-grounded model recommended raising prices on the existing tenant by 5-10% before renewal, an approach based on what actually worked for similar businesses in similar situations.
Another example involved an egg supplier where one customer represented 70% of revenue and one vendor represented nearly all costs. The frontier model recommended raising prices by 15-20%, which risked losing most of the revenue base. The grounded model instead recommended negotiating with the vendor for a 5-10% cost reduction, addressing the constraint-aware solution on the cost side rather than the risky revenue side.
Intuit’s research revealed what they term the “fluent bluff” - generic, fluent, and confident answers that frontier LLMs provide about money based on what they learned from internet content, blogs, books, and advice columns, rather than what actually happened in real business situations. Their analysis found that across the 100,000+ business scenarios, 40% of frontier model advice essentially recommended acquiring new customers, with an additional 14% suggesting increasing revenue from existing products. Over half of all advice boiled down to “acquire new customers and increase revenue,” reflecting textbook thinking rather than context-aware, outcome-verified guidance.
This finding aligned with recent research from Princeton that attempted to answer whether leading models could drive long-horizon business decisions. Researchers gave models a simulation environment with tools, data, and everything needed to make decisions across 500 days, starting each with one million dollars. Most models drove the simulated companies to bankruptcy before reaching 500 days. Remarkably, a simple rules-based system outperformed almost all the frontier models. Even the few models that generated profit did so only in specific instances, and when rerun, they also resulted in bankruptcy.
Intuit developed a sophisticated multi-stage approach that combines the strengths of frontier models with specialized training on verified outcomes. The architecture consists of three primary components working together in production.
The foundation begins with what Intuit calls “business trajectories” - detailed temporal representations of business states, actions, and outcomes extracted from millions of businesses across their product ecosystem. A business state represents a comprehensive snapshot at a given point in time, derived from financial data including general ledgers, profit and loss statements, cash flow statements, and invoices. From this rich data, they derive actions such as marketing campaign investments, payroll changes, hiring costs, pricing adjustments, and vendor negotiations. They then track outcomes across different time frames, measuring changes in profit, revenue, cash flow, or combinations thereof.
This extraction process creates millions of state-action-outcome vectors that form the training dataset for their specialized models. The scale and richness of Intuit’s data assets prove crucial here - they have comprehensive financial records across QuickBooks, TurboTax, Credit Karma, and Mailchimp, providing visibility into diverse business types, sizes, and situations.
The second component involves reinforcement learning models trained to understand causal relationships between actions and outcomes. Rather than relying on naive correlation, Intuit employs sophisticated causal inference techniques to measure true impact. They use conditional average treatment effect methods to account for selection bias and natural business trajectory differences. For example, when evaluating the impact of price increases, they compare similar businesses with equal propensity to raise prices - one group that actually raised prices versus one that did not. This approach accounts for the fact that businesses capable of raising prices may already be more successful, isolating the actual causal impact of the action itself.
The reinforcement learning model learns which actions lead to the best outcomes given specific business states and contexts. This model serves as a verification and selection layer in the production system.
The third component is a custom-trained LLM that generates the actual advisory content. Intuit took a mid-sized, more affordable model and trained it specifically to produce advice grounded in the reinforcement learning model’s understanding of verified outcomes. This training transforms generic textbook advice into evidence-based recommendations reflecting what actually works.
In production, Intuit employs a hybrid architecture that leverages both frontier models and their specialized grounded models. Frontier models are used for hypothesis generation - producing candidate actions that a business might take. These candidates benefit from the broad knowledge and reasoning capabilities of large models. However, these candidates are then filtered and evaluated through Intuit’s reinforcement learning model, which determines which suggestions would actually lead to positive outcomes versus mistakes that could harm the business.
This architecture recognizes the strengths and limitations of different model types. Frontier models excel at generating creative, contextually relevant suggestions and explaining concepts in natural language. However, they lack the domain-specific outcome verification that only comes from observing real-world results at scale. By combining hypothesis generation from frontier models with outcome verification from specialized models, Intuit creates a system greater than the sum of its parts.
The custom LLM then generates the final advisory output, explaining not just what the business should do, but why, grounded in evidence from similar businesses. This approach addresses both the accuracy problem and the trust problem - users receive not just better advice, but advice with transparent reasoning based on verifiable patterns.
Intuit conducted rigorous head-to-head testing of their custom model against all leading frontier models. Despite being a mid-sized and cheaper model, their approach outperformed frontier models across financial advisory tasks. This result challenges the common assumption that bigger models automatically deliver better results. The speaker emphasized that the moat is not model access but rather the unique data assets and the methodology for extracting verified outcome patterns from that data.
This evaluation approach represents an important LLMOps principle: domain-specific performance metrics matter more than general capabilities. A smaller model with domain-specific grounding can outperform much larger general-purpose models on specialized tasks, particularly when those tasks require understanding cause-and-effect relationships that aren’t directly learnable from text corpora alone.
Intuit deployed this technology as an AI business advisor currently in beta as a research preview with customers. The system proactively raises opportunities for businesses at specific points in time, providing recommendations with detailed explanations. The interface allows users to understand who is similar to them, what similar businesses did, and why the specific recommendation applies to their situation. Users can drill down into the reasoning and create actionable plans.
The speaker emphasized that great advisory experiences require two elements beyond just sound science. First, the system must understand user preferences - what they like, what they don’t like, their risk tolerance, and their business philosophy. Second, the experience must make users feel they are part of the decision-making process, creating trust rather than simply issuing directives. This human-centered design approach recognizes that even technically correct advice will fail if users don’t trust or understand it.
Several key LLMOps principles emerge from this case study. First is the distinction between context and experience. Even providing a frontier model with complete access to a business’s financial data - all the general ledgers, P&L statements, cash flows, and invoices - still represents just one data point. The model lacks the experiential learning that comes from observing many businesses in similar situations taking different actions and experiencing different outcomes. This mirrors how human advisors develop expertise: not from reading textbooks alone, but from years of observing what actually works in practice.
Second, measuring experience proves technically challenging. Intuit had to develop sophisticated causal inference approaches to extract meaningful signal from observational data. Simple before-and-after comparisons don’t work because businesses that take certain actions may be systematically different from those that don’t. Proper causal estimation requires comparing similar businesses with similar propensities to take actions, then isolating the treatment effect.
Third, outcome verification at scale represents a unique advantage that cannot be easily replicated. Intuit’s system of record across millions of small and medium businesses provides visibility into diverse situations, actions, and outcomes that no amount of internet text can capture. This data represents proprietary intelligence about what actually works in business, not just what people write about or believe works.
Fourth, the production architecture demonstrates thoughtful decomposition of the advisory task into components that leverage different model strengths. Rather than expecting a single model to do everything, Intuit uses frontier models for creative hypothesis generation, specialized RL models for outcome verification, and custom LLMs for generating grounded explanations. This modular approach allows optimization of each component and graceful degradation when individual components fail.
Fifth, the case illustrates the importance of domain-specific model development even in the era of powerful foundation models. The assumption that ever-larger general-purpose models will solve all problems proves false in domains requiring causal reasoning about outcomes. Investment in custom model training, specialized datasets, and domain-specific evaluation remains worthwhile even as frontier models advance.
The speaker positioned this work as part of a broader shift toward outcome-driven AI. The question is no longer which model is better in general, but how to steer AI to achieve desired outcomes in specific domains. Whether building anti-fraud systems, healthcare applications, logistics optimization, or developer tools, the winners will be those with the best systems of record, who can create unique datasets from those records and train models to achieve specific outcomes.
This represents a maturation of the field beyond the initial excitement of general-purpose models. While frontier models democratized access to impressive language capabilities, competitive advantage in production applications increasingly comes from domain-specific grounding, verified outcomes, and specialized training. Organizations with proprietary data assets showing what actually works in their domains possess advantages that cannot be easily replicated through model scale alone.
Intuit’s advantage stems from their comprehensive financial data ecosystem. QuickBooks provides detailed business financial records, TurboTax offers tax filing patterns, Credit Karma contributes consumer credit data, and Mailchimp adds marketing and customer engagement information. This multi-faceted view of business operations across millions of customers creates unprecedented visibility into business trajectories over time.
The scale proves crucial - approximately 100,000 business situations and timeframes analyzed, representing millions of state-action-outcome vectors. This scale enables statistically rigorous causal inference and training of models that can handle diverse business types, situations, and contexts. Smaller datasets would struggle to provide sufficient examples of similar situations for robust pattern recognition.
While the presentation focused on successes, several implicit challenges emerge. First, causal inference from observational data remains inherently difficult, even with large-scale data. Unmeasured confounders, selection effects, and time-varying factors all complicate the extraction of true causal relationships. Intuit’s approach using conditional average treatment effects helps but doesn’t eliminate these challenges entirely.
Second, the validation question remains complex. How does Intuit verify that their grounded model’s advice actually works prospectively rather than just fitting historical patterns? The beta deployment as a research preview suggests ongoing validation, but the full evaluation methodology isn’t detailed. Real-world advisory systems face the challenge that business outcomes depend on many factors beyond the actions recommended, making attribution difficult.
Third, the system must handle novelty - situations that don’t closely match any historical patterns in the training data. The speaker didn’t address how the system behaves when encountering edge cases or unprecedented situations where outcome-based grounding provides little guidance.
Fourth, maintaining these systems over time requires continuous data pipeline operation, model retraining, and monitoring. Business environments change, economic conditions shift, and what worked historically may not work in new contexts. The operational complexity of keeping outcome-grounded models current likely exceeds that of simply using off-the-shelf models.
The speaker concluded with actionable guidance for practitioners: look for places in your data where you can observe situations across multiple entities and verify outcomes. This pattern applies beyond finance - any domain with repeated situations, varied actions, and measurable outcomes offers opportunities for outcome-driven AI. Healthcare treatment decisions, logistics routing choices, content recommendation strategies, and many other domains exhibit this structure.
The emphasis on “verified outcomes in your data” highlights that the key LLMOps differentiator isn’t model architecture or training techniques per se, but rather the unique data assets and methodology for extracting causal patterns from those assets. Organizations should inventory their data for these outcome-rich scenarios before investing heavily in custom model development.
The case study ultimately demonstrates that production LLM systems for high-stakes domains require more than just prompting or fine-tuning frontier models with domain context. They require fundamentally different architectures that combine creative language generation with rigorous outcome verification, backed by proprietary data assets showing what actually works in practice. This represents a mature approach to LLMOps that goes beyond the initial wave of RAG and prompt engineering to tackle harder problems of causality, verification, and trust in domains where advice quality has real consequences.
OpenAI presented Agent RFT (Agent Reinforcement Fine-Tuning), a platform that enables organizations to fine-tune reasoning models to improve agentic behavior through real-time tool interactions and custom reward signals. The platform addresses the challenge of training AI agents that need to interact with external tools and environments during production workflows, moving beyond traditional supervised fine-tuning approaches. Multiple enterprise customers across coding, healthcare, and finance domains demonstrated significant improvements, including reduced tool call latency (up to 18% faster), elimination of long-tail loops (from 100+ messages to tight clusters), and substantial accuracy gains (5-23% improvements) while maintaining or reducing resource consumption through reinforcement learning-based credit assignment.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
The UK Met Office partnered with AWS to automate the generation of the Shipping Forecast, a 100-year-old maritime weather forecast that traditionally required expert meteorologists several hours daily to produce. The solution involved fine-tuning Amazon Nova foundation models (both LLM and vision-language model variants) to convert complex multi-dimensional weather data into structured text forecasts. Within four weeks of prototyping, they achieved 52-62% accuracy using vision-language models and 62% accuracy using text-based LLMs, reducing forecast generation time from hours to under 5 minutes. The project demonstrated scalable architectural patterns for data-to-text conversion tasks involving massive datasets (45GB+ per forecast run) and established frameworks for rapid experimentation with foundation models in production weather services.
