AWS SMGS faced significant business intelligence challenges including time-intensive manual data preparation, fragmented data across multiple systems, and limited dashboard accessibility that delayed critical leadership decisions. To address these issues, they built NarrateAI, an AI-powered conversational assistant using Amazon Bedrock AgentCore that delivers on-demand business insights through natural language. The solution employs a two-layer architecture combining batch narrative generation with real-time multi-agent orchestration, powered by Anthropic's Claude Sonnet 4. Since deployment, NarrateAI has served over 4,000 active users across AWS leadership, reducing business review preparation time from hours to minutes while maintaining comprehensive data accuracy and row-level security through persona-based narrative isolation.
NarrateAI represents a production-scale implementation of conversational AI for business intelligence within AWS’s Sales, Marketing and Global Services (SMGS) organization. The system was designed to solve critical operational challenges facing AWS leadership, including time-intensive manual data preparation that consumed hours before business reviews, data fragmentation across multiple dashboards and systems that created inconsistent metrics, and limited accessibility requiring specialized knowledge to navigate complex business intelligence tools. The solution demonstrates a mature approach to deploying LLMs in production with careful attention to accuracy, security, latency, and organizational scalability.
The NarrateAI architecture is structured around two distinct but complementary layers that separate concerns between data processing and real-time interaction. This architectural decision reflects a pragmatic understanding of production LLM deployment challenges, specifically around latency, cost, and accuracy tradeoffs.
The first layer handles automated narrative generation through batch processing. This layer operates on a scheduled basis to pre-generate comprehensive, persona-based narratives for each authorized user. The pipeline consists of three stages: data extraction using configuration-driven SQL templates that query Amazon Redshift with parameterized queries adapted to each user’s role and permissions; data transformation through AWS Lambda functions that convert structured data into standardized JSON using section-type logic including objects, arrays, breakdowns, and containers; and narrative rendering using Jinja templates to produce human-readable text from the structured data. A particularly noteworthy aspect is the hierarchical, business domain-aware chunking strategy that handles large datasets efficiently. Each user’s narrative is stored as an isolated text file in Amazon S3, implementing row-level security through complete data isolation at the storage layer rather than relying solely on query-time access controls.
The second layer provides the conversational AI interface for real-time user interaction. This layer is built on Amazon Bedrock AgentCore, which serves as the foundational orchestration infrastructure. When a user submits a natural language question through either Amazon Quick (a conversational interface) or the Web UI, the request flows to Amazon Bedrock AgentCore Runtime for processing. The architecture leverages AgentCore’s native capabilities for serverless deployment, built-in authentication, memory management, and foundation model integration, which the team reports accelerated deployment from months to weeks.
At the heart of the real-time layer sits the NarrateAI Agent, which implements a sophisticated multi-agent system coordinated by a Supervisor Agent. This supervisor orchestrates six specialized tools that work together to process queries:
The Question Classification tool analyzes incoming queries to determine user intent and complexity. Simple queries like “What is my team’s revenue this quarter?” are routed through a fast path for immediate resolution, while complex multi-part questions such as “Compare my top 5 accounts’ growth rates across all product lines” are automatically decomposed into parallel sub-tasks. This routing decision has significant implications for both response latency and computational cost.
The Persona Knowledge Identifier determines which narrative file should be retrieved from Amazon S3 based on the user’s role, organizational level, and data permissions. This component implements the security boundary that ensures users only access data they are authorized to view.
The Knowledge Extractor implements a table-of-contents (TOC) based retrieval approach rather than scanning entire narrative files. This design choice reflects an optimization for latency as narratives grow in size. By pre-analyzing document structure during batch processing and creating a navigable TOC, the system can selectively retrieve only relevant sections at query time.
The Relevancy Evaluator assesses extracted content against the user’s question to filter out tangential information and identify the most pertinent sections. This step helps reduce the context window sent to the LLM, which improves both response quality and cost efficiency.
The Answer Generator synthesizes coherent natural language responses from the relevant data, powered by Anthropic’s Claude Sonnet 4 accessed through Amazon Bedrock. The team chose Claude Sonnet 4 for its natural language understanding, complex business reasoning capabilities, and response generation quality.
The Online Evaluator represents a critical production safety mechanism. Every response is validated before delivery through cross-referencing generated numbers against source data, checking logical coherence, and confirming that claims are grounded in the user’s narrative. This addresses the well-known challenge of LLM hallucination, particularly problematic in business intelligence contexts where numeric accuracy is essential for decision-making.
The system uses Amazon Bedrock to access Anthropic’s Claude Sonnet 4 as its primary foundation model. The team highlights that Bedrock’s model flexibility allows them to test and upgrade to newer model versions without architectural changes, providing a clear upgrade path as more capable models emerge. This architectural decision to abstract the foundation model behind Bedrock’s API demonstrates forward-thinking LLMOps design that anticipates model evolution.
Safety and guardrails are implemented through Amazon Bedrock Guardrails with three custom-configured filters. Content filtering blocks inappropriate or harmful language. PII redaction helps prevent accidental exposure of sensitive personally identifiable information, which is particularly important given the business context where customer and employee data might be discussed. Tone guardrails ensure every response maintains a professional voice appropriate for AWS leadership, addressing the challenge of consistent brand voice in generative AI applications.
The architecture explicitly limits LLM involvement in numeric calculations to reduce hallucination risk. This design decision reflects a mature understanding that LLMs should not be relied upon for precise mathematical operations, and that deterministic calculation systems should handle numeric operations with the LLM focusing on natural language understanding and generation.
The team migrated conversation history (short-term memory) from a custom Amazon DynamoDB-based solution to AgentCore’s native memory capabilities. This migration eliminated the need to maintain custom session management code, simplified the architecture by consolidating memory management within AgentCore’s serverless infrastructure, and enabled faster feature development with built-in session handling. This evolution demonstrates the value of managed services for reducing operational overhead in LLMOps deployments, though it also represents a tradeoff in control and customization.
AgentCore Observability provides detailed insights into end-to-end agent session execution through integration with Amazon CloudWatch for monitoring and alerting. The team specifically highlights that traceability capabilities through OpenTelemetry built into AgentCore have improved debugging efficiency, with troubleshooting time dropping from tens of minutes or hours to single-digit minutes. This represents a significant operational improvement for production LLM systems where debugging can be particularly challenging due to the non-deterministic nature of model outputs and the complexity of multi-step orchestration workflows.
The knowledge engine relies on Amazon S3 for storing persona-specific narrative files, with AWS Lambda handling serverless data transformation. Amazon Redshift serves as the underlying data warehouse. The Data Refresh Scheduler automates periodic updates to ensure narratives reflect the latest business data. This batch-oriented approach to knowledge preparation trades freshness for query-time performance and cost efficiency—a reasonable tradeoff for business intelligence use cases where data typically updates on daily or weekly cycles rather than real-time.
The custom retrieval system using a TOC approach avoids the need to scan entire files at query time. This optimization becomes increasingly important as the knowledge base grows, and represents an alternative to traditional vector-based retrieval systems. The team’s choice suggests that for their structured business narrative use case, hierarchical document navigation may be more effective than semantic similarity search.
The team identifies four critical engineering challenges they addressed to achieve production readiness, providing valuable insights into real-world LLMOps concerns.
Accuracy and trust required implementing validation at multiple levels. Since LLMs can generate answers that sound correct but contain errors—a particularly serious risk for business decision-making—every response is validated before delivery by checking that numbers are accurate, metrics are consistent, and the answer is logically sound. The architecture limits LLM involvement in numeric calculations and uses automated checks to flag inappropriate language before responses reach users. This multi-layered validation approach demonstrates that production LLM systems cannot rely solely on the model’s output and must implement deterministic verification mechanisms.
Context awareness across organizational hierarchies presented the challenge of ensuring different leaders see appropriate data based on their roles and permissions. The system automatically identifies who is asking and what data they are permitted to see, then scopes every response to their role and reporting structure without requiring manual filters. This is implemented through the persona-based narrative generation approach, where security is enforced at data processing time rather than query time.
Response latency at scale required optimization as early versions experienced significant delays on complex questions. The system now routes questions based on complexity, with straightforward queries answered quickly and complex multi-part questions handled through parallel processing. The team notes that response times vary based on use case, query nature, and underlying data sources. Pre-analyzing document structure during data ingestion further reduces information retrieval latency per query. This tiered approach to query processing represents a practical solution to the latency challenges inherent in LLM-based systems.
Scope and topic coverage across multiple business domains (sales, finance, marketing) each with distinct terminology and logic required close collaboration with domain experts. The team developed a standardized template approach to encode institutional knowledge, reducing what once took months of custom development to weeks. This highlights the importance of domain expertise and structured knowledge engineering in production LLM deployments, and suggests that purely unstructured approaches may not suffice for enterprise business intelligence.
Row-level security is implemented through user-specific narrative generation rather than query-time filtering. Each user’s narrative file in Amazon S3 is fully isolated, helping prevent cross-user data leakage across the organization. User permissions are applied during data processing in the batch layer. This “security by construction” approach where unauthorized data never enters the user’s knowledge base represents a more robust security model than relying on filtering at query time, though it does require maintaining separate narratives for each user and role combination.
The system reports serving more than 4,000 active users across AWS leadership from CEO to regional managers, with business review preparation time reduced from hours to minutes. Leaders report increased confidence in data-driven decisions and improved ability to explore business performance through natural conversation rather than dashboard navigation.
However, several caveats warrant attention. The blog post is promotional material from AWS marketing its own services, so the claimed benefits should be viewed with appropriate skepticism. The text notes that “response times vary based on use case, query nature, and underlying data sources” without providing specific latency numbers, making it difficult to assess performance objectively. The system’s reliance on batch-generated narratives means there is inherent latency between when data updates and when users can query the new information. The TOC-based retrieval approach may not handle all types of cross-cutting questions as effectively as more sophisticated retrieval methods. The validation mechanisms, while multiple, still depend on the accuracy of the source data and the effectiveness of the validation logic itself.
The team outlines plans to evolve NarrateAI from a reactive conversational assistant into a proactive autonomous agent that delivers insights, surfaces anomalies, and suggests actions before leaders ask. The envisioned system would be triggered by business calendars, real-time data changes, and organizational context, powered by enhanced Amazon Bedrock AgentCore orchestration, event-driven architecture, and advanced multi-agent systems including Knowledge, Forecast, and Policy Agents. The system would deliver predictive analytics with safe, explainable autonomous actions governed by a full policy and audit layer. This vision represents the natural evolution from question-answering to proactive intelligence, though implementing such capabilities while maintaining accuracy, trust, and safety at scale presents significant additional challenges.
The architecture patterns demonstrated have potential applicability across industries beyond AWS’s internal use case. These include customized knowledge retrieval combining retrieval accuracy with generative flexibility, hierarchical document processing for intelligent navigation of complex knowledge bases, role-based access control for secure data access in multi-tenant environments, and parallel agent processing for responsive performance at scale. The separation of batch data processing from real-time query handling is a pattern that could apply to many enterprise AI deployments where data freshness requirements allow for scheduled updates rather than real-time synchronization.
The case study demonstrates a mature approach to production LLM deployment that balances AI flexibility with rule-based accuracy, enforces data access at the source, and optimizes for common use cases while reliably handling complex ones. The migration from custom infrastructure to managed services like AgentCore illustrates the ongoing evolution in the LLMOps ecosystem toward higher-level abstractions that reduce operational burden.
Notion, a knowledge work platform serving enterprise customers, spent multiple years (2022-2026) iterating through four to five complete rebuilds of their agent infrastructure before shipping Custom Agents to production. The core problem was enabling users to automate complex workflows across their workspaces while maintaining enterprise-grade reliability, security, and cost efficiency. Their solution involved building a sophisticated agent harness with progressive tool disclosure, SQL-like database abstractions, markdown-based interfaces optimized for LLM consumption, and a comprehensive evaluation framework. The result was a production system handling over 100 tools, serving majority-agent traffic for search, and enabling workflows like automated bug triaging, email processing, and meeting notes capture that fundamentally changed how their company and customers operate.
This panel discussion features three AI-native companies—Delphi (personal AI profiles), Seam AI (sales/marketing automation agents), and APIsec (API security testing)—discussing their journeys building production LLM systems over three years. The companies address infrastructure evolution from single-shot prompting to fully agentic systems, the shift toward serverless and scalable architectures, managing costs at scale (including burning through a trillion OpenAI tokens), balancing deterministic workflows with model autonomy, and measuring ROI through outcome-based metrics rather than traditional productivity gains. Key technical themes include moving away from opinionated architectures to let models reason autonomously, implementing state machines for high-confidence decisions, using tools like Pydantic AI and Logfire for instrumentation, and leveraging Pinecone for vector search at scale.
Toyota Motor North America (TMNA) and Toyota Connected built a generative AI platform to help dealership sales staff and customers access accurate vehicle information in real-time. The problem was that customers often arrived at dealerships highly informed from internet research, while sales staff lacked quick access to detailed vehicle specifications, trim options, and pricing. The solution evolved from a custom RAG-based system (v1) using Amazon Bedrock, SageMaker, and OpenSearch to retrieve information from official Toyota data sources, to a planned agentic platform (v2) using Amazon Bedrock AgentCore with Strands agents and MCP servers. The v1 system achieved over 7,000 interactions per month across Toyota's dealer network, with citation-backed responses and legal compliance built in, while v2 aims to enable more dynamic actions like checking local vehicle availability.
