## Overview This case study provides extensive insights into how four major technology and consulting companies (Autodesk, KPMG, Canva, and Lightspeed) have implemented AI agents in production environments using the Relevance AI platform. The discussion spans the entire lifecycle from initial experimentation through production deployment, offering valuable perspectives on LLMOps practices including governance, monitoring, evaluation, and change management. ## Company Contexts and Initial Approaches **Autodesk** serves customers across architecture, engineering, construction, and media production with design and engineering software. Their lifecycle automation team focused on delivering personalized value at scale, identifying AI agents as a tool to supercharge processes close to revenue generation, specifically around trial-to-conversion optimization. **KPMG** established an AI Lab within their KPMG Futures division to focus on cutting-edge research and development. They work with enterprise clients to pilot and prototype AI agents quickly, particularly for complex multi-agent systems that clients struggle to implement themselves. Their focus is on back-office and knowledge work across functions like finance, risk, compliance, and procurement. **Canva** took an early AI-first approach at the company level, providing education and tools to all employees while building a deliberate go-to-market roadmap for AI efficiency gains. Their goal was freeing up customer-facing teams to spend more time with customers rather than on administrative tasks. **Lightspeed Commerce** provides unified commerce point-of-sale and payment platforms for retailers, restaurants, and golf courses. Their performance marketing team adopted a "crawl, walk, run" approach to agent deployment, starting with limited volume and extensive human oversight before scaling. ## Platform Selection and Technical Architecture A recurring theme across all organizations was the need to move beyond open-source frameworks like Crew AI and Autogen. While these frameworks offered deep capability and control, they required too much development time to meet business needs for rapid prototyping and value demonstration. KPMG specifically noted that clients don't want to wait weeks or months to determine commercial viability. The key differentiators that led to platform selection were the ability to rapidly prototype multi-agent systems, maintain traceability and explainability of agent collaboration, implement human-in-the-loop approval workflows, and integrate with existing enterprise systems. Canva emphasized the importance of avoiding tool proliferation, noting that customer-facing reps already navigate seven to eight different platforms. The ability to surface AI capabilities within existing workflows in Slack and Salesforce rather than adding new interfaces was critical. ## LLMOps Practices: From Prototype to Production ### Process Mapping and Knowledge Embedding Lightspeed demonstrated a rigorous approach to embedding domain expertise into agents. They worked with subject matter experts from sales, marketing, technical teams, security, privacy, and legal to design agents that fit within their ecosystem. The team identified top performers within specific niches, interviewed them to understand their processes and success factors, then created detailed process maps showing both current human workflows and desired agent workflows. This process map served as the source of truth for agent builds and stakeholder alignment. KPMG emphasized that successful automation requires first articulating processes well, then understanding why the process exists and what acceptable outcomes look like. Many businesses have created processes reactively when things go wrong but have never clearly defined what "good" looks like. AI agents expose this gap, forcing organizations to establish first principles around process outcomes. ### Training and Iteration Approaches Autodesk treated AI agents like any workforce that requires training on company knowledge, brand voice, tone, and KPIs. Their institutional knowledge and industry playbooks were considered core competitive advantages that needed to be modularized into shared knowledge bases accessible to agents. This required extensive collaboration with product marketing teams who conducted user behavior research and business development teams who provided feedback on what messaging worked. The iterative process involved breaking down complex prompts into separate agents that work together, creating a distributed AI workforce rather than relying on a single monolithic prompt. Autodesk emphasized there is no magical prompt that fixes everything, and prompts age as the world changes, requiring constant updates and treating agents like products rather than one-time implementations. ### Crawl, Walk, Run Deployment Strategy Lightspeed implemented an exemplary phased rollout approach for their closed-lost research and outreach agent, which hands qualified leads to human sales reps. The deployment phases included: In the crawl phase, they ran the agent at very low volume with human approval of all research reports generated by the agent. This allowed them to validate research quality and identify failure modes without risk. During the walk phase, they continued approvals but expanded to email generation and email replies, using both the platform's built-in approval functionality and Slack-based approvals for mobile accessibility. The team continuously refined outputs, making small changes directly in the platform or escalating strategic changes to wider stakeholder groups. In the run phase, they moved toward autopilot but maintained ongoing monitoring to ensure outputs met quality standards. This phased approach was essential given they were reaching out to actual customers, making the risk of poor quality outputs significant. KPMG noted that this gradual ramp-up helps identify issues with both internal processes and automated systems, not just the agents themselves. The slow scaling allows teams to document and fix problems in the surrounding ecosystem that the agent depends on. ### Human-in-the-Loop and Quality Assurance Human oversight played a critical role across all implementations. Lightspeed heavily relied on Relevance AI's human-in-the-loop approval functionality, accessible through both the platform and Slack. The ability to review and approve agent runs while mobile was particularly valued. The approval gates served multiple purposes beyond quality control. They built confidence among stakeholders by demonstrating the agent could perform consistently before removing oversight. They provided data for iterative refinement, showing exactly where agents fell short. They also created an audit trail for compliance and governance, particularly important when dealing with customer communications. Autodesk emphasized that AI agents are only as good as the humans guiding them, requiring close monitoring of results combined with regular feedback from business development, sales, customer support, and customer success teams. This feedback loop informed prompt iterations and process adjustments. ## Data Strategy and Integration ### Data Quality and Enrichment Data readiness emerged as a fundamental challenge. KPMG reported that early client projects failed to see benefits because clients didn't have data ready for agents to review and process. Rather than waiting until data is perfect, KPMG now uses AI agents to perform data diagnostics, identifying gaps and priorities, then uses the tools themselves to create or harvest data to the required state. Canva implemented sophisticated enrichment workflows using AI agents, going beyond basic data hygiene to build account databases that reflect their very specific go-to-market edge cases and data sources. This allowed them to identify and score accounts in ways that off-the-shelf enrichment vendors couldn't support, giving them creative license to shape their go-to-market in a Canva-specific way. ### System Integration Architecture The technical implementations required extensive integration work. Lightspeed noted they had hundreds of sales reps globally and a diverse tech stack, creating complexity where agents had to fit within human processes and automated systems based on triggers while working with incomplete contextual data. Canva emphasized platform thinking over point solutions, seeking to integrate AI capabilities into existing surface areas rather than adding new tools. Their approach involved surfacing agents in Slack and Salesforce where reps already work, avoiding the proliferation of new interfaces that would compete for attention. The integration strategy also involved careful consideration of data residency and processing location. KPMG highlighted that many Australian clients have policies prohibiting information processing outside Australia, requiring local model hosting and specific contractual arrangements with AI providers. ## Governance, Security, and Compliance ### Multi-Stakeholder Alignment Lightspeed created an "AI pod" structure that brought together subject matter experts from communications, content, marketing automation, information security, and other teams. This cross-functional squad approach ensured all perspectives were represented in agent design and deployment decisions. The governance process involved ruthlessly prioritizing use cases due to high excitement and demand across the organization, then getting clear alignment from stakeholders on priorities before entering execution mode. Managing expectations was identified as a key challenge given the rapid pace of AI advancement and the tendency for proof-of-concepts to generate enthusiasm. ### Security and Privacy Controls Foundational security controls mentioned across implementations included role-based access control to prevent unauthorized users from accessing sensitive use cases or API keys, monitoring and observability with comprehensive audit logs to track user actions within systems, and scoped integration access allowing certain integrations for specific use cases without exposing them company-wide. KPMG emphasized that transparency in monitoring enables senior leaders to be confident that AI is being used appropriately and producing aligned outputs. This visibility into agent behavior and performance is critical for building organizational trust. Canva's IT team focused on creating friction-free paths from idea to production while maintaining appropriate safeguards. This involved thinking ahead about what types of data would be allowed in different platforms, what use cases would be enabled, and how to make the process as seamless as possible for business units. ### IP and Data Processing Concerns KPMG identified two primary client concerns that consistently emerged. The first was data domiciling and processing location, with policies often requiring all information processing to remain within specific geographic boundaries. This sometimes limited access to the latest models given later release schedules in markets like Australia. The second concern involved intellectual property protection and trust in model providers. Even with contractual agreements that data won't be used for model retraining, some organizations remained skeptical. KPMG addressed this through offline models completely disconnected from cloud providers and enterprise agreements with AI providers offering guarantees on data usage and storage. ## Measurement and ROI ### Defining Success Metrics Autodesk emphasized the importance of clear, actionable north star metrics rather than broad goals. Instead of simply "increase pipeline," they focused on specific, measurable objectives like "increase trial to conversion rate for these specific products." This narrower scope allowed focused iteration on initial models, with successful components later replicated to other products while addressing areas falling short. Lightspeed as a performance-focused growth marketing team had to adapt their measurement approach to include labor productivity metrics, which were new territory. They combined traditional performance marketing metrics with agent-specific metrics and labor productivity measures, building custom reporting to demonstrate value and support scaling decisions. ### Business Impact Measurement The measurement approach involved tracking multiple data points including engagement rates post-agent deployment, pipeline generation and increase, conversion rates at various funnel stages, response rates to agent-generated communications, and time savings enabling reps to spend more time with customers. Autodesk saw direct data increases in engagement after AI agents began personalizing content at scale. They received positive feedback from business development and sales teams, who reported smoother conversations because prospects were better educated and nurtured when reaching the sales queue. Operational teams saw efficiencies that enabled more meaningful work. Canva focused on how agents could help achieve customer journey objectives like closing business faster, onboarding customers faster, and freeing up rep time for customer engagement. Their framework treated AI initiatives like any other product development cycle with clear business cases and ROI justification. ## Organizational Change Management ### Cultural Enablement Canva's approach to AI adoption was heavily influenced by leadership intent and company mission. Their founders established a mission-driven culture focused on empowering the world to design, with a two-step plan of becoming as valuable as possible to do as much good as possible. This mission-driven approach, combined with the principle of "give value first, get value second," created a culture where fear wasn't a motivator and experimentation was encouraged. The company held an AI Discovery Week providing three days of training followed by a two-day hackathon for all employees. This demonstrated executive commitment but was complemented by department-level initiatives tailored to how each team works. For IT, this meant a Monday "hour of power" for ongoing AI enablement, while other departments adopted different cadences. ### Building Internal Expertise Canva created a dedicated go-to-market AI lead role to drive the agent strategy. Prior to this role, there was significant appetite and evolving roadmaps but difficulty making sustained impact without someone focused solely on overcoming blockers. The dedicated resource enabled faster shipping of more complex AI-driven workflows beyond basic quick wins. Lightspeed's approach involved empowering a senior AI marketing technologist who had been experimenting with AI for performance marketing since before ChatGPT's release. The company supported stretch projects and development, creating an environment encouraging experimentation and learning. ### Stakeholder Communication Success required bringing teams along the journey rather than treating them as bystanders. Autodesk emphasized that when they showed lift and results to cross-functional teams, collaboration came naturally as people saw the true potential. Maintaining shared analytics and KPIs across teams kept everyone invested in the journey. KPMG found that quick wins building credibility were essential. Early, low-complexity implementations that saved reps fifteen minutes daily or helped with email follow-ups built momentum and appetite for more sophisticated solutions. Starting with something that provided visible value to end users created the foundation for larger transformations. ## Technical Implementation Patterns ### Multi-Agent Architectures The implementations moved beyond single-agent systems to distributed agent workforces. Autodesk described breaking complex prompts into separate agents that collaborate, finding this approach better delivered their brand voice and messaging while achieving business objectives. KPMG's focus on complex multi-agent systems stemmed from client needs that went beyond simple use cases. The ability to have multiple agents collaborating with clear control over that collaboration, combined with traceability and explainability of the interactions, was a core requirement. ### Modular Knowledge Bases Autodesk modularized their institutional knowledge and best practice playbooks into shared databases accessible to multiple agents. This involved extensive collaboration with product marketing for user behavior research and ongoing feedback from business development and sales teams about what worked. The modular approach allowed knowledge to be reused across different agent implementations while maintaining consistency in how the company's expertise was represented. Updates to knowledge bases could propagate across all agents using that information. ### Reusable Components As implementations matured, teams began extracting reusable components from successful agents. Lightspeed mentioned decoupling the research component from their outreach agent to provide it as a standalone resource to sales teams. This pattern of identifying successful agent capabilities and making them available in new contexts accelerated subsequent agent development. ## Lessons Learned and Best Practices ### Start Small, Learn Fast, Scale Deliberately The consistent recommendation across all organizations was to start with focused use cases close to revenue with clear objectives rather than attempting broad transformation. Autodesk advised finding a data point as a starting point that's close to revenue with a clear job to support conversion, emphasizing this focused beginning even though broader success is achievable. KPMG recommended picking a few things and doing them really well, combining quick wins with transformative projects in a deliberate roadmap. Treating implementations with the respect given to full product development cycles, starting with business objectives and working backward, provides structure for sustainable scaling. ### Expect and Plan for Iteration Lightspeed emphasized that things usually don't go as planned, with roadblocks requiring adjustment. They favored quality over speed despite stakeholder pressure to move quickly, finding this patience led to better outcomes. The need for continuous iteration extended beyond initial deployment. Autodesk noted that prompts age as the world and society change, requiring constant updates. The process never reaches a static 100% completion point but rather requires ongoing product-like treatment. ### Balance Experimentation with Production Rigor The tension between rapid experimentation and production quality was a recurring theme. KPMG noted it's easy to prototype but very hard to get to production, particularly at enterprise scale. Giving teams time to tinker and iterate while maintaining urgency created the right balance. Canva's approach of supporting both quick wins and longer-term transformative bets allowed them to build momentum while working toward substantial changes. The quick wins provided proof points and learning while larger initiatives received proper planning and execution. ### Integrate, Don't Proliferate The strong emphasis on integration over new tools reflected real challenges with technology sprawl. Canva explicitly sought platforms that could bring AI automation into existing workflows rather than adding new interfaces for already-overwhelmed users. This integration mindset extended to processes as well as tools. Agents needed to fit within existing human workflows and system triggers rather than requiring wholesale process redesign. ## Future Outlook and Predictions KPMG predicted that business-to-business communications will increasingly become agent-to-agent interactions, with invoices, contract negotiations, and routine communications handled autonomously. Human involvement will focus on exceptions and high-value interactions. They also anticipated mass adoption of agents in sales cycles leading to commoditization, where automated outreach becomes so prevalent it loses meaning, fundamentally changing what the sales process becomes. Canva's perspective focused on how AI shortens decision timelines and increases efficiency while potentially shifting the balance between seller-led and buyer-led transactions. Buyers may take more agency using AI tools to engage with vendors on their own terms. The common vision involved ambient agents working in the background for research, information compilation, and outreach, then handing off to human teams for what they do best: customer conversations, relationship building, and complex problem-solving. The goal is automating administrative and preparatory work so humans can focus on distinctly human activities requiring empathy, intuition, and relationship development. ## Critical Success Factors Several factors emerged as critical for successful AI agent implementation in production. Organizations need executive sponsorship and clear mission alignment to overcome fear-based resistance and enable experimentation. Close partnership between IT, security, and business units is essential to enable safe, rapid deployment. Investment in process documentation and articulation of desired outcomes provides the foundation for effective automation. Phased rollouts with human oversight build confidence while identifying and addressing issues. Continuous monitoring and feedback loops enable ongoing refinement and quality assurance. Focus on integration with existing tools and workflows avoids change fatigue and adoption barriers. Clear ROI frameworks combining productivity metrics with business outcomes justify continued investment. The implementations described represent mature approaches to LLMOps, demonstrating how enterprises can move beyond experimentation to derive real value from AI agents while maintaining appropriate governance and risk management.