## Overview Intuit has implemented one of the largest scale LLM deployments in the financial services sector through their TurboTax platform, serving 44 million tax returns annually. The company developed a comprehensive AI assistant called "Intuit Assist" to help customers understand their tax situations, deductions, credits, and overall refund outcomes. This implementation represents a sophisticated LLMOps architecture built on their proprietary platform called GenOS (Generative OS), designed specifically to handle the unique challenges of operating LLMs in highly regulated environments at massive scale. The primary business driver was to increase customer confidence in tax filing by providing clear, accurate explanations of complex tax situations. Rather than just automating tax calculations, the system focuses on education and explanation, helping users understand why they receive certain deductions or credits and what their overall tax outcome means. This approach addresses a critical pain point in tax software where users often complete their returns without truly understanding the underlying tax implications. ## Technical Architecture and Platform Intuit built GenOS as their foundational generative AI platform, recognizing that off-the-shelf LLMOps tooling was insufficient for their specific requirements around safety, security, and regulatory compliance. The platform consists of several key components that work together to orchestrate the entire AI experience: **GenUX** serves as the user interface layer, providing the frontend experience for customers interacting with the AI assistant. This component handles the presentation of AI-generated explanations and manages the conversational flow between users and the system. **Orchestrator** acts as the central routing and coordination layer. Given that different teams at Intuit work on various AI components and solutions, the orchestrator determines which specific AI service should handle each incoming query. This is crucial because different types of questions require different approaches - some need access to proprietary tax engines, others require general tax knowledge, and some need real-time calculation capabilities. The system architecture supports both static and dynamic query types, each optimized for different use cases. Static queries are essentially prepared statements for common tax explanation scenarios, such as explaining the components of a tax refund (deductions, credits, withholdings, etc.). These queries are pre-engineered and optimized for consistent, reliable responses. Dynamic queries handle ad-hoc questions from users about their specific tax situations, such as "Can I deduct my home office expenses?" or more unusual scenarios. ## Model Selection and Multi-Model Strategy Intuit employs a sophisticated multi-model approach, primarily leveraging Anthropic's Claude family and OpenAI's GPT models. The company has established a multi-million dollar contract with Anthropic, making them one of Claude's largest enterprise customers. For production workloads, they primarily use Claude for the core tax explanation functionality, while GPT-4 Mini has been deployed for question-and-answer scenarios, though they continuously evaluate newer model versions as they become available. The model selection process is driven by several factors: accuracy for tax-specific content, latency requirements, cost considerations, and regulatory compliance capabilities. The team noted that model upgrades are not as straightforward as one might expect - even upgrading within the same vendor's model family (such as moving from Claude Instant to Claude Haiku) requires significant evaluation and validation work. **Fine-tuning Implementation**: Intuit has experimented with fine-tuning Claude 3 Haiku through AWS Bedrock, focusing specifically on static query scenarios. The primary motivation for fine-tuning was to reduce prompt size and improve latency while maintaining quality. In tax scenarios, prompts can become extremely large as they incorporate complex user tax situations with multiple income sources, deductions, and family circumstances. By fine-tuning models on their specific use cases, they aimed to achieve the same quality with more concise prompts. The fine-tuning process adheres to strict data privacy regulations, using only consented user data and operating within regulatory frameworks like SOC 2 Type 2 compliance. However, the team found that while fine-tuned models performed well on quality metrics, they became somewhat too specialized for their specific use cases, potentially limiting flexibility for handling edge cases or evolving requirements. ## Knowledge Integration and RAG Systems The system incorporates multiple knowledge sources through sophisticated RAG (Retrieval Augmented Generation) implementations. Intuit maintains proprietary tax knowledge engines that have been developed and refined over many years, containing detailed information about tax laws, regulations, and calculations. These engines serve as the authoritative source for numerical calculations and tax determinations. **Traditional RAG** is used to incorporate general tax information, IRS documentation, and Intuit's proprietary tax guidance. This system helps answer questions that require reference to tax codes, regulations, or procedural information. **Graph RAG** has been implemented for more complex scenarios requiring understanding of relationships between different tax concepts, entities, and regulations. The team reports that Graph RAG significantly outperforms traditional RAG for providing personalized, contextually relevant answers. This is particularly important in tax scenarios where the interaction between different tax situations (employment income, investment income, deductions, credits, family status, etc.) creates complex relationships that benefit from graph-based reasoning. The knowledge systems are continuously updated to reflect annual changes in tax laws and IRS regulations. Tax year transitions (such as from 2023 to 2024) require significant updates to ensure the AI system reflects current tax rules and procedures. ## Safety and Regulatory Compliance Operating in the highly regulated tax domain requires extensive safety measures and compliance frameworks. Intuit has implemented multiple layers of safety guardrails to prevent the system from providing incorrect tax advice or hallucinating numerical information. **Calculation Safety**: The system strictly separates LLM-generated explanations from actual tax calculations. All numerical computations (tax amounts, refunds, deductions) come from Intuit's proven tax knowledge engines, not from LLMs. This architectural decision prevents the AI from hallucinating tax numbers, which could have serious legal and financial consequences for users. **Content Filtering**: Machine learning models operate as safety layers to detect and prevent potentially problematic content before it reaches users. These models specifically check for hallucinated numbers, inappropriate advice, or content that doesn't align with established tax guidance. **Regulatory Compliance**: The system operates under various regulatory frameworks including SOC 2 Type 2, with careful attention to data privacy and security requirements. Only consented user data is used for model training and improvement, and all processing occurs within approved, secure environments. ## Evaluation Framework and Quality Assurance Intuit has developed a comprehensive, multi-phase evaluation system that represents one of the most sophisticated LLMOps evaluation frameworks described in the industry. The evaluation system addresses three core quality dimensions: accuracy, relevancy, and coherence. **Human Expert Integration**: Tax analysts and domain experts play a central role in the evaluation process. Rather than having data scientists and ML engineers write prompts, Intuit uses tax experts as prompt engineers. This approach ensures that prompts reflect accurate tax knowledge and appropriate professional guidance. The domain experts provide initial manual evaluations that become the foundation for automated evaluation systems. **Manual Evaluation Phase**: During development, tax experts manually review LLM outputs against established tax guidance and professional standards. This creates a "golden dataset" of high-quality examples that serves as the benchmark for automated systems. **LLM-as-Judge Systems**: For scaled evaluation, Intuit has implemented LLM-as-judge systems that can automatically evaluate new outputs against the standards established by human experts. These systems use GPT-4 series models to assess whether responses meet quality standards. The LLM judges operate based on carefully crafted prompts that incorporate guidance from tax experts and reference the golden dataset. **Continuous Monitoring**: Production systems include automated monitoring that samples real user interactions and evaluates response quality in real-time. This allows the team to detect quality degradation or emerging issues quickly. **Automated Prompt Engineering**: The team has developed internal tooling for automated prompt optimization, which helps improve and update the LLM-as-judge systems as requirements evolve. The evaluation framework proved essential when migrating between model versions, as it provided objective measures to ensure new models maintained or improved quality standards. ## Production Challenges and Operational Lessons **Latency Management**: Unlike traditional backend services with 100-200 millisecond response times, LLM services operate at 3-10 second latencies. During peak tax season (around April 15th tax deadline), these latencies can increase significantly as prompt sizes balloon with complex user situations and system load increases. The team has designed the product experience around these latency constraints, implementing appropriate fallback mechanisms and user interface patterns that manage user expectations. **Vendor Lock-in and Contracts**: The team highlighted the significant challenge of vendor relationships in the LLM space. Large-scale deployments require substantial financial commitments through long-term contracts, creating vendor lock-in effects. Beyond financial considerations, the team noted that prompts themselves create a form of technical lock-in, as migrating between providers requires significant re-engineering even when the underlying capabilities are similar. **Model Lifecycle Management**: Frequent model updates and releases from providers create ongoing operational overhead. Each model change, even within the same provider's family, requires comprehensive evaluation and validation. The team emphasized that model upgrades are not trivial operations and require the same rigor as major system changes. **Scalability Architecture**: Supporting 44 million tax returns requires careful attention to scalability patterns. The team uses different AWS accounts and environments for testing and production, with platform teams providing infrastructure support. The orchestration layer helps distribute load across different model endpoints and services based on query types and system capacity. ## Integration with Existing Systems The LLM-powered assistant integrates deeply with Intuit's existing tax platform rather than operating as a standalone system. The tax knowledge engines that power TurboTax's core functionality provide the authoritative data for calculations and determinations, while the LLM layer adds explanation and education capabilities. This integration strategy allows Intuit to leverage their decades of investment in tax software while adding modern AI capabilities. Users receive the same reliable tax calculations they've always received, enhanced with AI-powered explanations that help them understand and trust the results. ## Future Directions and Ongoing Development The team continues to evaluate new models and approaches, with ongoing experiments in various areas. They maintain internal model development capabilities alongside their use of commercial LLM providers, allowing them to explore hybrid approaches. The evaluation framework they've built provides the foundation for safely testing and deploying new capabilities as they become available. The emphasis on continuous evaluation and improvement reflects the dynamic nature of both the tax domain (with annual regulatory changes) and the LLM technology landscape (with frequent model updates and new capabilities). Their systematic approach to evaluation and deployment provides a model for other organizations operating LLMs in regulated environments at scale.