## Case Study Overview This case study examines the large-scale deployment of AI assistants across multiple enterprise companies, primarily focusing on Payfit (a payroll and HR software company with 16,000 clients) and Alan (a health insurance company covering 500,000 people across France, Spain, and Belgium). The case is presented through a panel discussion featuring Gabriel (CEO of Dust.tt), Charles (CTO of Alan), and Silva (Chief Product Officer at Payfit), providing insights into real-world LLMOps implementation at scale. Dust.tt serves as the underlying platform enabling these companies to build, deploy, and manage AI assistants that integrate with their existing tech stacks. The platform provides access to multiple LLM providers (OpenAI, Anthropic, Google, Mistral) while offering tools for retrieval-augmented generation (RAG), multi-tool assistance, and workflow automation. ## Technical Architecture and Implementation The core technical approach centers around a meta-platform strategy that abstracts underlying LLM complexity while providing enterprise-grade integration capabilities. Rather than building individual AI solutions for each tool (Notion AI, Slack AI, Google Duets), the companies opted for a unified platform that could access information across all their existing systems simultaneously. The technical implementation relies heavily on retrieval-augmented generation (RAG) as a foundational component. Charles from Alan specifically mentions that "the hard problem about a tool like dust is the rag part" and acknowledges that while they could have built this capability internally, it would have consumed significant engineering resources without providing competitive advantage. The RAG implementation enables the AI assistants to access and synthesize information from various company data sources in real-time. The platform supports multi-agent workflows where different assistants can work together in chains. Silva from Payfit describes implementing "assistants that work together" for customer support, including resolution assistants and investigation assistants that collaborate based on confidence scores and complexity assessment. This multi-agent approach allows for more sophisticated workflows that can handle varying levels of complexity and expertise requirements. Model selection and switching represents a critical operational consideration. The companies actively monitor and switch between different LLMs based on performance characteristics. As of the discussion, Claude 3.5 Sonnet was identified as the current top performer, with the speakers noting the rapid pace of model improvements making it "alarming" to keep up with developments. The platform's abstraction layer allows for seamless model switching without requiring changes to individual assistants. ## Production Use Cases and Workflows ### Sales Process Automation Alan implemented a comprehensive sales workflow automation that demonstrates the power of chained AI assistants. The process begins with an account summary assistant that analyzes all available data about a sales prospect and compresses it into a digestible briefing for sales representatives. Following sales meetings, transcription services feed into meeting note generation assistants that automatically populate CRM systems. Finally, follow-up email assistants generate personalized communications based on the meeting content. This end-to-end automation reportedly increased sales productivity by approximately 20% while improving data quality and consistency. The standardized format of AI-generated meeting notes enabled the marketing team to extract intelligence across hundreds of sales interactions, creating a virtuous cycle of improved sales and marketing alignment. ### Customer Support Enhancement Payfit developed a sophisticated customer support system using paired assistants that handle tickets ranging from simple access issues to complex payroll interpretations requiring 20 years of expertise. The system implements a confidence scoring mechanism that routes tickets between resolution and investigation assistants based on complexity assessment and available knowledge sources. The multi-tiered approach acknowledges the reality that customer support requires different levels of expertise, and the AI system can appropriately triage and escalate issues while maintaining high accuracy standards. This represents a mature approach to LLMOps that considers both technical capabilities and business requirements. ### Performance Management and HR Processes Both companies highlighted performance review assistance as one of their most successful use cases. The AI assistants help managers synthesize employee performance data, draft comprehensive reviews, and maintain consistency across evaluations. Charles specifically mentioned being "super grateful" for AI assistance during performance review week, noting it saved him significant time that would otherwise have come from sleep hours. This use case demonstrates how AI can augment rather than replace human judgment in sensitive HR processes, providing structure and efficiency while maintaining the personal touch required for meaningful performance conversations. ## Deployment Strategy and Adoption Framework The companies implemented a dual approach combining top-down leadership commitment with bottom-up discovery and adoption. Leadership established clear vision and organizational support while empowering individual teams to discover and develop relevant use cases. ### Organizational Structure The rollout involved dedicated AI champions (typically 3-5 people initially) who worked across teams to identify opportunities and provide training. These champions combined technical AI knowledge with deep understanding of specific business functions, enabling them to bridge the gap between technological capability and practical application. The adoption strategy included team-by-team presentations (25-50 people at a time) that educated users about AI capabilities while providing concrete examples relevant to their specific functions. For HR teams, this might include bias-free job description writing; for design teams, it could involve persona development and SEO optimization. ### Metrics and Success Measurement The companies achieved impressive adoption rates, with Alan reporting 80% of employees using AI weekly and Payfit reaching 30-40% adoption. The measurement approach balances quantitative metrics for high-impact use cases with qualitative feedback for experimental applications. For critical use cases like customer support, teams aim for approximately 20% productivity improvements with careful measurement. For other applications, they rely on user feedback and subjective assessments, recognizing that over-measuring can stifle innovation and experimentation. ## Challenges and Technical Considerations ### Model Reliability and Consistency One significant challenge involves the non-deterministic nature of LLM outputs. The speakers acknowledge that while creating impressive demos is straightforward, ensuring consistent performance in production requires careful prompt engineering and robust testing frameworks. Charles emphasized the importance of making demos "cool" while ensuring that real-world use cases work "very consistently" to maintain user confidence and adoption. ### Security and Compliance Framework Security considerations extend beyond traditional data protection to include new risks specific to LLMs. Charles notes that "llms are really bad at lying" and can potentially expose sensitive information if not properly controlled. However, he also dispels common misconceptions about data retention and retraining risks, explaining that the probability of models regurgitating specific company data is "close to zero." The companies implemented frameworks that allow teams to safely experiment with AI while maintaining compliance with regulations like GDPR. This involves both technical controls and user education about appropriate use cases and data handling. ### Documentation and Knowledge Management The AI implementation revealed significant gaps in existing documentation processes. Silva notes that AI adoption "helps us discover a lot of gaps in our documentation processes" as teams realize the importance of well-structured, accessible information for AI systems to function effectively. This creates a positive feedback loop where AI implementation drives improvements in knowledge management practices, which in turn enhance AI system performance and organizational learning. ### Platform Fragmentation and Standardization As AI adoption spreads organically across organizations, managing fragmentation becomes increasingly important. Teams may develop similar assistants independently, leading to duplicated effort and inconsistent approaches. The companies are working to balance creative experimentation with practical standardization and knowledge sharing. ## Build vs. Buy Decision Framework Both companies initially leaned toward building custom AI solutions but ultimately chose the platform approach based on several factors: **Cost Considerations**: Building and maintaining RAG infrastructure, model management, and integration capabilities would require significant ongoing engineering investment without providing competitive differentiation. **Flexibility Requirements**: Single-vendor AI solutions (like Notion AI or Slack AI) create ecosystem lock-in, while meta-platforms enable information synthesis across all organizational tools. **Resource Constraints**: In the current economic environment, both companies prioritized resource allocation toward core business functions rather than AI infrastructure development. **Speed to Market**: The platform approach enabled rapid experimentation and deployment across multiple use cases without lengthy development cycles. ## Financial and Business Impact The productivity improvements achieved through AI implementation are substantial and measurable. Both companies report baseline productivity gains of approximately 20%, with some specific use cases achieving 50% or higher improvements. Charles suggests that in certain highly automatable tasks, the gains could reach 400% through team size reduction and task elimination. Beyond direct productivity metrics, the companies observe qualitative improvements in employee satisfaction as tedious tasks are automated, allowing workers to focus on more meaningful and strategic activities. This creates a "collapse of the talent stack" where smaller teams can accomplish the work previously requiring larger groups. The financial impact extends to customer acquisition costs for sales teams and operational efficiency improvements across multiple departments. The standardization of AI-generated outputs also enables better analytics and business intelligence across previously disparate workflows. ## Future Outlook and Evolution The speakers express strong optimism about continued AI advancement while acknowledging that current deployment and cultural adoption lag behind technological capability. Even if model development froze today, they believe significant untapped potential exists in better deploying existing technology across organizational workflows. Vision capabilities represent an emerging frontier, with both companies testing multimodal applications for document processing, website analysis, and user interface interaction. While current accuracy rates (88-95%) aren't yet sufficient for error-sensitive applications, the rapid improvement trajectory suggests broader adoption within 2-3 years. The discussion reveals a mature understanding that AI success depends more on organizational change management and thoughtful implementation than on raw technological capability. The most successful deployments combine technical sophistication with deep understanding of business processes, user needs, and change management principles. This case study demonstrates that successful enterprise AI deployment requires a holistic approach encompassing technology selection, organizational design, security frameworks, adoption strategies, and continuous iteration based on real-world feedback and evolving capabilities.