AWS Sales faced an agent proliferation challenge with over 20 domain-specific AI agents deployed globally, forcing sales representatives to manually navigate between systems and manage context across fragmented conversations. To address this, AWS built Field Advisor on Amazon Bedrock AgentCore, creating a unified conversational interface that orchestrates specialized agents, maintains context, and handles human-in-the-loop workflows. The solution delivered measurable results including over 120K prompts processed, up to 2 hours saved per week per sales rep, 41% reduction in latency, and consolidation from seven AWS accounts to a single AgentCore Runtime.
AWS Sales built Field Advisor, an agentic AI orchestration platform powered by Amazon Bedrock AgentCore, to address the operational challenges created by having more than 20 specialized AI agents deployed across their global sales organization. This is a production-scale LLMOps case study that demonstrates how AWS addressed real enterprise challenges around agent proliferation, context management, secure multi-tenant execution, and continuous quality monitoring. The case study is particularly valuable because it reveals both the architectural decisions and the production infrastructure patterns needed to operate multi-agent systems at enterprise scale.
The core problem was cognitive overhead: sales representatives needed to know which agent to invoke for each task (CRM operations, meeting scheduling, customer insights, product recommendations, compliance checks), manually manage context across fragmented conversations, and combine outputs from different systems. This context-switching consumed time that should have been spent on customer conversations. Field Advisor addresses this by serving as a central orchestration layer that routes requests to specialized agents while maintaining a single conversational interface.
Field Advisor processes over 120,000 prompts across all modalities since launch, demonstrating genuine production adoption. The human-in-the-loop component specifically saves large-scale sales representatives up to 2 hours per week, time that is redirected to customer conversations and strategic planning. The migration to Amazon Bedrock AgentCore delivered three key operational improvements: a 41% reduction in latency compared to previous infrastructure, consolidation from seven separate AWS accounts to a single AgentCore Runtime, and the removal of custom-built systems for memory, observability, and authentication. These metrics demonstrate real LLMOps efficiency gains rather than just theoretical benefits.
From a balanced perspective, the case study is clearly promotional content for AWS services, but the technical details and specific metrics suggest genuine production deployment. The quoted user testimonials reference concrete use cases (creating CRM tasks from meeting notes, validating 450 accounts, processing pricing requests) rather than vague benefits, which adds credibility. However, readers should note that the 2-hour-per-week savings applies specifically to “large-scale sales representatives” with the human-in-the-loop component, not necessarily all users across all workflows.
The architecture is built on Amazon Bedrock AgentCore, which provides the foundational capabilities for production agentic AI: isolated execution environments through MicroVMs for secure multi-tenant operations, a unified gateway for tool and agent access across AWS accounts, persistent memory for session and long-term context, consistent identity propagation with OAuth integration, built-in observability across complex request flows, and integrated evaluation for continuous quality monitoring.
When a sales rep submits a question through supported channels (CRM system, Slack, or standalone web portal), the request passes through an authentication service that verifies identity and issues an OAuth token. AgentCore Runtime receives the authenticated request, initializes a supervisor agent built with Strands Agents, and manages the full execution lifecycle. The supervisor agent analyzes the query and determines how to handle it by invoking local tools for CRM lookups and knowledge base retrieval, calling remote MCP tools for systems integrated through Model Context Protocol, or delegating to specialized domain agents running in separate AgentCore Runtimes. AgentCore Identity propagates the authenticated identity to every downstream tool and agent, ensuring consistent security context. Results are synthesized into a unified response and streamed back through AgentCore Runtime’s native streaming support.
The architecture follows a supervisor-subagent pattern where the supervisor acts as the primary orchestrator and specialized domain agents (for compliance, product recommendations, meeting scheduling, etc.) are integrated as tools. Each remote agent is defined with a name, description, and a single query parameter. When the supervisor determines a question should be handled by a domain agent, it invokes the corresponding tool, which handles the full invocation lifecycle: constructing the payload, authenticating through OAuth, sending the request to the remote AgentCore Runtime, and streaming back the response. The supervisor sees a flat tool catalog and the Strands reasoning loop handles orchestration, meaning adding a new agent requires only a configuration entry with no changes to orchestration logic.
The supervisor agent uses the latest Anthropic Claude model available on Amazon Bedrock as the foundation model, accessed through the Amazon Bedrock cross-Region inference profile for higher throughput. To reduce latency on multi-turn conversations, the team built a custom PromptCachingBedrockModel that extends Strands’ BedrockModel to add incremental prompt caching. Before each model call, the implementation removes any previous cache points from the conversation history and adds a fresh one to the latest message, so everything up to the most recent turn is cached and the model only processes new content on each turn.
This is a practical LLMOps optimization that directly addresses the latency challenges of maintaining long conversation histories. The implementation carefully manages cache point placement because Amazon Bedrock limits the number of cache points per request and requires non-decreasing TTL values across the request (tools, system prompt, then messages). The result is that each new user turn extends the cached prefix incrementally rather than reprocessing the full conversation, contributing to the overall 41% latency reduction.
The Strands Agents hook system is central to how agent behavior is customized without modifying the core reasoning loop. Hooks are registered for several cross-cutting concerns: error circuit breaking automatically skips tools that fail repeatedly within a single invocation; contingent authorization intercepts tool calls that access sensitive CRM data, pauses execution using Strands Interrupts to request user consent, and only proceeds after approval; write confirmation for tools that mutate data presents proposed changes and requires explicit approval before execution; citation extraction processes knowledge base retrieval results to extract source references and attach them to the final response; and tool call streaming streams tool invocation status to the front end so users see real-time progress as the agent works.
This hook-based approach is an important LLMOps pattern because cross-cutting concerns like authorization and error handling are decoupled from tool implementations. Teams that own remote agents don’t need to implement circuit breaking or streaming—the supervisor handles it uniformly. The case study notes that for builders designing similar systems, hooks are a natural extension point that avoids the complexity of middleware chains or decorator stacks. This represents a production-tested pattern for managing complexity in multi-agent systems.
Field Advisor implements human-in-the-loop workflows using the native Interrupt feature in Strands Agents. When a tool or remote agent needs user input (for example, acknowledging a data access agreement before querying CRM records), execution pauses and the interrupt is returned to the client. The user responds through Slack or the CRM interface, and the agent resumes with their input. This pattern was extended to work across agent boundaries: when a remote agent returns an interrupt in its response payload, the local tool wrapper detects it and raises a Strands Interrupt on behalf of the remote agent.
This design choice is noteworthy from an LLMOps perspective because it avoids building a separate human-in-the-loop service with message queues and polling. Because Strands Interrupts persist state in the session manager, the pattern is purely code-based with no additional infrastructure components needed. The key insight is that the local tool wrapper acts as a bridge, translating the remote agent’s interrupt response into a native Strands Interrupt so the framework handles all pause/resume mechanics. This architectural decision reduces operational complexity while maintaining consistent user experience across all agents in the system.
The human-in-the-loop component maintains control over critical business actions while accelerating routine tasks. When Field Advisor needs to update CRM data, it presents proposed changes and waits for explicit approval before making modifications. This helps maintain data accuracy, prevents unintended or harmful changes, and maintains accountability by keeping sales reps in control of critical updates. One user testimonial describes Field Advisor creating CRM tasks from a meeting notes file with one-click approvals, saving at least 15 minutes for a single customer interaction.
AgentCore Memory provides both short-term and long-term persistence. Short-term memory maintains the full conversation history within a session, while long-term memory uses a semantic strategy to store summaries and important facts across sessions so the agent remembers user preferences and prior interactions even in new conversations. The implementation uses a custom session manager that extends the Strands Agents AgentCoreMemorySessionManager to optimize write patterns by syncing state after each invocation rather than after every message, which reduces API calls without sacrificing durability.
For context management, the implementation uses the Strands Agents SummarizingConversationManager to handle conversations that approach the model’s token context window. Older messages are summarized to open up context space while preserving key information. Token limits are also enforced on user input, and large tool responses are truncated before passing them back to the model, preventing context overflow from a single source. This multi-layered approach to context management demonstrates production-ready patterns for handling long-running conversations at scale.
Sales reps can upload documents and images directly within conversations. Strands accepts file bytes as ContentBlock inputs, supporting text formats and images. When a file is uploaded, its content becomes part of the conversation history and is available to the agent for analysis, summarization, or answering follow-up questions. For unsupported file types such as JSON, the system extracts the text content and prepends it to the user message. This improves upon the previous solution, which was limited to five files per chat with a combined 10 MB size limit, demonstrating iterative improvement based on production usage patterns.
Field Advisor supports three categories of tools through a unified interface. Local tools run directly inside the AgentCore Runtime for CRM queries, knowledge base retrieval, and data lookups. Remote MCP tools connect to external systems via the Model Context Protocol—on startup, the runtime establishes connections to registered MCP servers through AgentCore Gateway, fetches their tool definitions, and creates corresponding local Strands tools. More than 20 MCP tools have been onboarded this way, each integrated in minutes rather than days. Remote agents in other AgentCore Runtimes are wrapped as tools with the same interface, making the distinction between a tool and an agent transparent to the model.
The onboarding process for MCP tools is straightforward: register the MCP configuration in the Registry service, and the supervisor automatically discovers it on the next startup with no code changes or redeployment of the runtime. This operational pattern is significant for LLMOps because it enables continuous expansion of capabilities without disrupting the production system. The unified approach means adding a new capability (whether it’s a local function, an MCP server, or a full agent) requires no changes to the orchestration logic.
Field Advisor also supports a pass-through mode where sales reps explicitly select a specific agent to handle their request. In this mode, the request bypasses the supervisor entirely—the runtime forwards the query directly to the target remote agent and streams its response back, preserving the sub-agent’s formatting and citations. Users can switch between the supervisor and pass-through mode within the same conversation. This flexibility acknowledges that while orchestration reduces cognitive load for most queries, domain experts sometimes benefit from direct access to specialized agents.
Strands integrates natively with OpenTelemetry, and AgentCore Observability captures the resulting traces, logs, and metrics. Every agent invocation, tool call, and model interaction is traced end-to-end, including calls that fan out to remote agents in other accounts. This distributed tracing capability is critical for debugging when a single user question can fan out to several remote agents and MCP tools in parallel. The observability data serves dual purposes: operational monitoring for detecting failures and performance degradation, and continuous evaluation through AgentCore Evaluations.
The team runs both online evaluation against live production traffic and offline evaluation against curated test sets. The built-in evaluators in Amazon Bedrock AgentCore continuously measure agent quality across dimensions including answer relevance, context precision, action correctness, and faithfulness. This provides a feedback loop that helps identify regressions and improve orchestration quality over time. From an LLMOps best practices perspective, the case study recommends that builders start with online evaluation against production traffic rather than building curated test sets first, because real user queries surface edge cases that synthetic tests miss, and the built-in evaluators in AgentCore Evaluations provide a useful baseline without custom metric development.
This evaluation approach reflects a mature understanding of LLMOps challenges. Building comprehensive test sets before production deployment is time-consuming and often misses real-world edge cases. Starting with online evaluation allows teams to capture actual usage patterns and prioritize evaluation efforts based on real user needs. The continuous monitoring enables detection of regressions before they significantly impact users, supporting the kind of rapid iteration needed for improving AI systems in production.
Security is layered across the solution at multiple levels. AgentCore Runtime’s MicroVM isolation provides each user session with its own secure execution context, preventing cross-contamination between user sessions in a multi-tenant environment. Amazon Bedrock Guardrails filter both user inputs and model responses to block harmful or out-of-scope content, providing a defense layer against prompt injection and inappropriate outputs. AgentCore Identity propagates authenticated user identity to every downstream system, ensuring consistent access control enforcement. Tools and remote agents enforce access controls consistently, including contingent authorization checks that require explicit user consent before accessing sensitive CRM data.
This multi-layered security approach demonstrates production-grade LLMOps practices for enterprise deployment. The identity propagation is particularly important because it maintains audit trails and access control enforcement across complex multi-agent flows that span multiple AWS accounts. The contingent authorization pattern ensures that even with orchestration and automation, human accountability is maintained for sensitive operations.
Field Advisor is embedded directly into the tools sales teams use daily, working inside CRM systems, Slack, and first-party applications. This embedded access removes workflow disruption and keeps reps focused on customer interactions rather than switching between systems. The architectural decision to meet users where they work, rather than requiring them to adopt a new standalone tool, likely contributed significantly to the adoption numbers (120K+ prompts processed).
The solution also includes proactive recommendations that complement conversational capabilities with push-based alerts through Slack. Sales reps receive AI-driven insights about customer service usage, business data trends, and CRM hygiene alerts. These insights enable proactive customer outreach rather than reactive responses, demonstrating that the system goes beyond query-response patterns to actively surface valuable information.
The migration to Amazon Bedrock AgentCore consolidated infrastructure from seven separate AWS accounts to a single AgentCore Runtime, removing custom-built systems for memory, observability, and authentication. This consolidation represents significant operational simplification—managing seven separate AWS accounts with custom infrastructure for cross-cutting concerns creates substantial overhead for maintenance, security updates, and troubleshooting.
The engineering team now focuses on product features that directly improve customer outcomes rather than infrastructure maintenance. This shift in engineering focus is one of the key value propositions of managed LLMOps platforms: reducing undifferentiated infrastructure work so teams can focus on domain-specific intelligence and user experience improvements. However, readers should consider that this consolidation benefit is specific to teams migrating from fragmented custom infrastructure to a unified managed service, rather than a universal benefit applicable to all deployment patterns.
The case study concludes with practical guidance for teams building similar multi-agent orchestration solutions. AWS recommends starting with a single supervisor agent on AgentCore Runtime, adding tools incrementally through AgentCore Gateway, and using Strands hooks to layer in cross-cutting concerns like authorization and error handling. This incremental approach lets teams validate each component in production before adding complexity, reducing risk and enabling faster iteration cycles.
This deployment guidance reflects lessons learned from building Field Advisor and represents valuable LLMOps wisdom: start simple, validate in production, and add complexity incrementally as needs become clear. The recommendation to use hooks for cross-cutting concerns rather than building them into individual tools demonstrates architectural maturity—separating concerns at the framework level rather than duplicating logic across components.
From a balanced LLMOps perspective, this case study demonstrates a sophisticated production deployment of multi-agent orchestration with genuine operational metrics. The 41% latency reduction, infrastructure consolidation from seven accounts to one, and specific time savings (2 hours per week for human-in-the-loop workflows) suggest real production optimization rather than purely theoretical benefits. The technical details around prompt caching, interrupt-based human-in-the-loop patterns, and hook-based orchestration customization represent production-tested patterns that other teams could apply.
However, readers should recognize several caveats. This is promotional content for AWS services, and the case study emphasizes Amazon Bedrock AgentCore capabilities without discussing tradeoffs, vendor lock-in considerations, or challenges encountered during implementation. The solution is specifically designed for AWS Sales workflows and may not generalize to all enterprise contexts. The time savings (2 hours per week) apply specifically to large-scale sales representatives using human-in-the-loop workflows, not necessarily all users. The case study doesn’t discuss total cost of ownership, development timeline, team size, or failure modes encountered during production operation.
The architectural patterns are valuable regardless of platform: supervisor-subagent orchestration, hook-based customization for cross-cutting concerns, interrupt-based human-in-the-loop workflows, incremental prompt caching for latency reduction, unified tool interfaces that abstract whether capabilities are local functions or remote agents, online evaluation against production traffic before building comprehensive test sets, and embedded deployment in existing user workflows rather than standalone tools. These patterns represent mature LLMOps thinking that teams could apply using various platforms.
The user testimonials reference specific use cases (validating 450 accounts efficiently, creating CRM tasks from meeting notes, retrieving customer information without system switching) that add credibility, though they’re also selectively chosen positive examples. The overall case study represents a strong example of production multi-agent orchestration at enterprise scale, with valuable technical details and architectural patterns, while readers should maintain awareness of its promotional nature and consider their own specific requirements and constraints when evaluating applicability.
Notion, a knowledge work platform serving enterprise customers, spent multiple years (2022-2026) iterating through four to five complete rebuilds of their agent infrastructure before shipping Custom Agents to production. The core problem was enabling users to automate complex workflows across their workspaces while maintaining enterprise-grade reliability, security, and cost efficiency. Their solution involved building a sophisticated agent harness with progressive tool disclosure, SQL-like database abstractions, markdown-based interfaces optimized for LLM consumption, and a comprehensive evaluation framework. The result was a production system handling over 100 tools, serving majority-agent traffic for search, and enabling workflows like automated bug triaging, email processing, and meeting notes capture that fundamentally changed how their company and customers operate.
Sierra has built an enterprise agent platform serving most of the Fortune 20 companies, focusing on customer experience across sales, service, and loyalty touchpoints. The platform addresses the challenge of building reliable, low-latency conversational agents that can handle complex customer interactions across voice and chat modalities in dozens of languages. Sierra's approach combines a constellation of 10-15 models per conversation turn, custom infrastructure for sensitive operations like payments (achieving PCI DSS level one certification), and a no-code journey builder that compiles to their Agent SDK. The company has achieved notable success with outcome-based pricing models where agents earn commissions on sales, demonstrating measurable business value through improved resolution rates, conversion rates, and customer satisfaction metrics across retail, airline, and other enterprise verticals.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
