AWS developed Account Plan Pulse, a generative AI solution built on Amazon Bedrock, to address the increasing complexity and manual overhead in their sales account planning process. The system automates the evaluation of customer account plans across 10 business-critical categories, generates actionable insights, and provides structured summaries to improve collaboration. The implementation resulted in a 37% improvement in plan quality year-over-year and a 52% reduction in the time required to complete, review, and approve plans, while helping sales teams focus more on strategic customer engagements rather than manual review processes.
AWS implemented Account Plan Pulse as a comprehensive LLMOps solution to transform their internal sales account planning process. The system addresses the challenges that emerged as AWS scaled globally, including inconsistent plan quality across regions and industries, resource-intensive manual review processes, and knowledge silos that prevented effective cross-team collaboration.
The core technical implementation leverages Amazon Bedrock as the foundation model service, enabling AWS to build a production-ready generative AI system that processes account plans at enterprise scale. The solution architecture follows a structured pipeline approach that begins with ingestion from their CRM system into Amazon S3 buckets through scheduled batch processing. This approach ensures continuous analysis of the most current account plan information without overwhelming the system with redundant processing.
The preprocessing layer operates at two distinct levels to optimize efficiency and quality. The first layer implements an ETL flow that organizes required files for processing, while the second layer performs input validation just before model calls. A particularly sophisticated aspect of their preprocessing is the use of plan modification timestamps to process only recently updated documents, significantly reducing computational overhead and costs. The system extracts text content from HTML fields and generates structured metadata for each document, transforming everything into a standardized Parquet format stored in S3.
The Amazon Bedrock integration performs two primary functions that demonstrate sophisticated LLMOps practices. First, the system evaluates account plans against 10 business-critical categories through 27 specific questions with tailored control prompts, creating what they call an Account Plan Readiness Index. This automated evaluation provides specific improvement recommendations rather than just scoring. Second, the system extracts and synthesizes patterns across plans to identify customer strategic focus and market trends that might otherwise remain isolated in individual documents.
A critical aspect of their LLMOps implementation is the use of structured output prompting with schema constraints. This ensures consistent formatting that integrates seamlessly with their reporting tools and downstream systems. The asynchronous batch processing architecture allows evaluation and summarization workloads to operate independently, improving system resilience and scalability.
One of the most technically sophisticated aspects of their production implementation is their approach to handling the non-deterministic nature of LLMs. AWS developed a statistical framework using Coefficient of Variation (CoV) analysis across multiple model runs on the same inputs. This framework addresses three critical challenges: output variability from identical inputs, the evolving nature of account plans throughout customer relationships, and different evaluation priorities across AWS teams based on customer industry and business needs.
The CoV analysis serves as a correction factor to address data dispersion, calculated at the evaluated question level. This scientific approach allows them to measure and stabilize output variability, establish clear thresholds for selective manual reviews, and detect performance shifts that require system recalibration. Account plans falling within established confidence thresholds proceed automatically through the system, while those outside thresholds are flagged for manual review.
Their validation framework demonstrates comprehensive LLMOps practices aligned with security best practices. They implement input and output validations following the OWASP Top 10 for Large Language Model Applications. Input validation includes necessary guardrails and prompt validation, while output validation ensures results are structured and constrained to expected responses. They’ve also implemented automated quality and compliance checks against established business rules, additional review processes for outputs that don’t meet quality thresholds, and feedback mechanisms that improve system accuracy over time.
The dynamic threshold weighting system represents another sophisticated LLMOps capability, allowing evaluations to align with organizational priorities by assigning different weights to criteria based on business impact. This enables customized thresholds across different account types, applying different evaluation parameters to enterprise accounts versus mid-market accounts. These business thresholds undergo periodic review with sales leadership and adjustment based on feedback, ensuring the AI evaluations remain relevant while maintaining quality standards.
From a technical infrastructure perspective, the solution utilizes Amazon S3 for secure storage of all processed account plans and insights, implements a daily run cadence that refreshes insights and enables progress tracking, and provides interactive dashboards offering both executive summaries and detailed plan views. This infrastructure supports the scale requirements of a global sales organization while maintaining security and compliance standards.
The production engineering aspects of their implementation address key challenges in enterprise LLM deployment. They’ve built reliability into their AI evaluations through statistical frameworks, handled the evolving nature of business documents through dynamic processing capabilities, and created flexibility for different organizational needs through configurable evaluation criteria. Their approach to measuring and controlling output variability through statistical analysis represents a mature approach to LLMOps that goes beyond basic prompt engineering.
Looking forward, AWS plans to enhance Pulse’s capabilities by connecting strategic planning with sales activities and customer outcomes, analyzing account plan narratives to identify new opportunities, and leveraging new Amazon Bedrock capabilities including flows for workflow orchestration, Guardrails for enhanced safety, agentic frameworks, and Strands Agents with Amazon Bedrock AgentCore for more dynamic processing flows.
The case study demonstrates several key LLMOps principles in action: systematic approaches to handling model non-determinism, comprehensive validation frameworks, integration with existing business workflows, statistical methods for quality control, and iterative improvement based on feedback. The reported results of 37% improvement in plan quality and 52% reduction in processing time suggest that their LLMOps implementation has delivered significant business value while maintaining the reliability and consistency required for enterprise sales processes.
Novartis partnered with AWS Professional Services and Accenture to modernize their drug development infrastructure and integrate AI across clinical trials with the ambitious goal of reducing trial development cycles by at least six months. The initiative involved building a next-generation GXP-compliant data platform on AWS that consolidates fragmented data from multiple domains, implements data mesh architecture with self-service capabilities, and enables AI use cases including protocol generation and an intelligent decision system (digital twin). Early results from the patient safety domain showed 72% query speed improvements, 60% storage cost reduction, and 160+ hours of manual work eliminated. The protocol generation use case achieved 83-87% acceleration in producing compliant protocols, demonstrating significant progress toward their goal of bringing life-saving medicines to patients faster.
Toyota Motor North America (TMNA) and Toyota Connected built a generative AI platform to help dealership sales staff and customers access accurate vehicle information in real-time. The problem was that customers often arrived at dealerships highly informed from internet research, while sales staff lacked quick access to detailed vehicle specifications, trim options, and pricing. The solution evolved from a custom RAG-based system (v1) using Amazon Bedrock, SageMaker, and OpenSearch to retrieve information from official Toyota data sources, to a planned agentic platform (v2) using Amazon Bedrock AgentCore with Strands agents and MCP servers. The v1 system achieved over 7,000 interactions per month across Toyota's dealer network, with citation-backed responses and legal compliance built in, while v2 aims to enable more dynamic actions like checking local vehicle availability.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
