## Overview This case study documents Salesforce's internal deployment of Einstein Copilot, their conversational AI assistant built on large language models (LLMs), within their own enterprise CRM environment. Published in April 2024, the article shares lessons learned from rolling out the AI assistant to thousands of internal employees starting in February 2024. The deployment represents a significant "dogfooding" exercise where Salesforce used their own product internally before broader customer availability, providing practical insights into enterprise LLMOps challenges and solutions. It's worth noting that this case study comes directly from Salesforce and is inherently promotional in nature. While the lessons shared are valuable, readers should consider that the narrative emphasizes successes and positions the product favorably. The claimed time savings of "countless hours" lacks specific quantification, and the 80% success rate target, while reasonable, leaves room for significant failure cases in production. ## Deployment Context and Scale The deployment occurred within Salesforce's internal org, which the article describes as exceptionally complex with thousands of objects, hundreds of thousands of fields, and billions of records. This scale presented significant testing challenges that smaller organizations might not face. The actual technical deployment of Einstein Copilot took less than two hours, but the comprehensive testing against this massive dataset took considerably longer, suggesting that organizations should budget significant time for validation activities rather than just technical implementation. The rollout strategy followed a phased approach, starting with just 100 sellers as a controlled pilot group. This small initial cohort was intentional—it enabled direct feedback collection and rapid iteration before expanding to thousands of users. This approach aligns with LLMOps best practices of gradual rollout and careful monitoring before broad deployment. ## Testing and Evaluation Methodology One of the most operationally significant aspects of this case study is the testing framework developed for Einstein Copilot. The team created a structured approach to evaluating LLM performance in production contexts: - For each standard Copilot action (like "draft an email"), they prepared approximately 30 potential user queries representing realistic use cases - They manually evaluated whether each query was successfully executed - They established an 80% success rate target for queries related to supported use cases - For the 20% of failing queries, they made deliberate decisions about whether to document as known limitations or build custom actions to address gaps - In total, they tested more than 250 queries, creating a foundational regression test set This query set became a critical asset for ongoing maintenance. Every time the team added a custom action or modified the configuration, they could run this test suite to detect regressions. The recommendation to add approximately 30 additional test queries for every new custom action provides a concrete heuristic for teams building similar systems. The 80% success rate target is an interesting choice—it acknowledges that LLM-based systems will not achieve perfect accuracy while setting a reasonable bar for production readiness. However, the article doesn't discuss how they handled the 20% failure cases in production or what user experience mitigations were in place for unsuccessful queries. ## Custom Actions and Extensibility A key technical aspect of the deployment involved extending beyond standard actions to build custom actions tailored to specific business needs. The standard actions covered common tasks like summarizing records, querying data, and drafting emails. However, user feedback revealed gaps that required custom development. Examples of custom actions developed include: - A task prioritization action that helps sellers identify which tasks to focus on - An action to surface major changes to records the user owns - Website browsing to enrich account data - Distilling multiple knowledge articles to generate answers about internal sales processes The article notes that many of these custom actions would eventually ship as standard actions in the product, suggesting a valuable feedback loop between internal deployment and product development. From an LLMOps perspective, this demonstrates the importance of having a flexible architecture that allows for rapid extension without requiring fundamental platform changes. The integration between Einstein Copilot and Salesforce's existing automation capabilities (Flow and Apex) is highlighted as a key architectural decision. The system allows custom actions to trigger auto-launched Flows, which can include decision nodes routing to various custom prompt templates. This hybrid approach—combining deterministic workflow automation with generative AI—represents a pragmatic production architecture that leverages LLMs where they excel (text generation, summarization) while relying on traditional logic for predictable operations. ## Trust Layer and Safety Considerations The article references the "Einstein Trust Layer" as part of the architecture, though details are sparse. The team made sure to communicate with executives about the limitations of generative AI upfront, explicitly stating that "it isn't a magic silver bullet" and warning about the likelihood of "unexpected outcomes." This expectation-setting is crucial for LLMOps success, as it prevents organizational disappointment and builds realistic understanding of LLM capabilities and limitations. The emphasis on being clear about limitations with stakeholders represents an important organizational LLMOps practice. Many AI deployments fail not because of technical issues but because of misaligned expectations. By frontloading these conversations, the team created space for iteration and improvement rather than immediate negative judgment. ## Metrics and Monitoring The team tracked several key metrics throughout the deployment: - Daily active users - Queries per session - User satisfaction While these metrics are mentioned, specific values or trends are not shared, which limits the ability to assess the actual quantitative impact. The claim of "saving countless hours" is vague and should be viewed skeptically without supporting data. The article mentions that Einstein Copilot includes event log functionality for tracking conversations, enabling analysis of query patterns at scale. This observability feature is essential for understanding actual user behavior versus intended use cases. The team used this data to identify unanswered queries and understand user expectations, feeding back into the custom action development process. ## Enablement and Change Management A substantial portion of the case study addresses user enablement, which the team identifies as "key" to success. Their enablement strategy included: - Short demo videos showing how sellers can use Copilot Actions within their workflows - A dedicated Slack channel with built-in workflows for capturing feedback - Additional documentation including step-by-step instructions and sample queries - Notifications through meetings and internal newsletters - Multi-channel communication to ensure broad awareness The emphasis on showing users what Einstein Copilot can and cannot do reflects a mature understanding of deploying AI tools. Users who don't understand system capabilities may either underutilize the tool or develop frustration from unmet expectations. The example of users asking about weather in San Francisco illustrates how users may have broader expectations than the system was designed to address. Recommended actions on record pages (account, contact, lead, opportunities, cases) were added to help users understand what questions and actions were available, reducing the "blank canvas" problem that can inhibit adoption of conversational interfaces. ## Continuous Improvement Model The article strongly emphasizes treating Einstein Copilot deployment as a continuous improvement project rather than a "deploy and forget" approach. Key practices include: - Holding regular sessions with small user groups for testing and feedback - Building lists of real user queries for testing and brainstorming new actions - Collecting queries on a biweekly basis from users - Monitoring monthly releases of new standard actions - Maintaining close collaboration between business (Global Seller Experience team) and IT (Business Technology team) This operational model acknowledges that LLM-based systems require ongoing attention and refinement. The biweekly feedback cadence and monthly product updates create a rhythm of continuous evaluation and improvement that aligns with modern DevOps and MLOps practices. ## Use Case Selection Criteria The team developed a framework for evaluating potential Einstein Copilot use cases, asking key questions: - Would Einstein Copilot improve the user experience? - Does the potential action fit within the product's user interface? - Will a conversational interface add value? - Is the new approach reducing clicks compared to existing workflows? - How conversational is the use case? Not all AI experiences need to live within a conversational interface, and the team deliberately routed some capabilities to other mechanisms (Flow automation, direct Trust Layer calls). This selective approach helps maintain the quality and coherence of the conversational experience rather than trying to force every feature through the chat interface. The guidance that LLMs are "great at generating summaries and texts" provides a useful heuristic for identifying strong use cases. The mobile voice-to-text capability is highlighted as a significant productivity enabler, reflecting the natural fit between conversational AI and voice interfaces. ## Critical Assessment While this case study provides valuable operational insights, several limitations should be noted. The article is inherently promotional and lacks specific quantitative results. Claims of "countless hours" saved are not substantiated with data. The 80% success rate target, while reasonable, means 1 in 5 queries may fail, which could significantly impact user trust and adoption at scale. The testing methodology, while structured, relies on manual evaluation which may not scale for larger deployments or more frequent changes. The article does not address challenges around cost management, latency considerations, or handling of sensitive data—all critical LLMOps concerns. Despite these limitations, the case study offers practical guidance for enterprises deploying conversational AI, particularly around phased rollout, systematic testing, user enablement, and continuous improvement processes. The lessons about starting simple, maintaining feedback loops, and setting realistic expectations are broadly applicable beyond Salesforce's specific implementation.