Stripe, processing $1.4 trillion annually across 50 countries, faced a critical compliance scaling challenge where skilled analysts spent up to 80% of their time navigating fragmented systems rather than performing risk assessments. To address this, Stripe built a production-grade AI agent system on AWS using Amazon Bedrock, implementing a ReAct agent framework with human-in-the-loop oversight, task decomposition via directed acyclic graphs (DAG), and a dedicated agent service infrastructure distinct from traditional ML inference systems. The solution achieved a 26 percent reduction in median review handling time while maintaining over 96 percent helpfulness ratings from reviewers, with human experts retaining final decision authority and full audit trails for regulatory compliance.
Stripe’s case study represents a sophisticated production deployment of agentic AI for financial compliance operations. The company processes approximately $1.4 trillion in annual payment volume across 50 countries, representing roughly 1.3 percent of global GDP, which positions it at a critical intersection of technological innovation and regulatory compliance requirements. The core challenge Stripe faced was how to scale compliance operations proportionally with business growth without corresponding headcount increases, while maintaining regulatory quality standards and full auditability.
The problem was particularly acute because skilled compliance analysts were spending up to 80% of their time navigating fragmented systems to gather documentation rather than performing high-value risk assessments. This represents a common pattern in regulated industries where the information-gathering overhead dominates expert time. Stripe’s solution leverages agentic AI to transform this dynamic, achieving a 26 percent reduction in median review handling time while maintaining over 96 percent helpfulness ratings and keeping human experts firmly in control of final decisions.
The technical implementation centers on three foundational pillars that inform all design decisions. First is oversight and accountability, implemented through human-centered validation with configurable approval workflows and multi-layered decision checkpoints. The system is explicitly designed so that humans stay in the driver’s seat, supported but not replaced by agents. Second is transparency, achieved through full audit trails with immutable documentation of every action, decision, and rationale. Third is efficiency, where pre-investigation and dynamic analysis allow deeper reviews at a faster pace without sacrificing quality.
A critical architectural insight Stripe developed is that assigning a single agent to handle long, complicated reviews in one pass wouldn’t work effectively. A single, unconstrained agent would focus too much on wrong aspects and not enough on what was actually needed. Instead, Stripe made the problem tractable through task decomposition, breaking complicated reviews into composable, bite-sized sub-tasks. Each sub-task can potentially depend on the results of other sub-tasks, forming a directed acyclic graph (DAG) structure. These “rails” serve multiple purposes: they verify each agentic process runs only on vetted questions where quality has been measured through quality testing, they confirm investigations cover required bases, and they provide agents with sufficient context and focus to deliver quality results.
Despite rigorous quality testing of agent responses in each sub-task, Stripe’s implementation crucially does not rely outright on agent responses. Instead, responses are provided as supplementary information to human reviewers, who must ultimately answer each sub-task of the review. This design solves for oversight and accountability while still capturing efficiency benefits. The orchestration approach means that as human reviewers answer questions, those validated responses can be used in prompts for deeper questions during the same review, creating a dynamic, context-aware investigation flow.
For fetching research on each sub-question, Stripe implemented a compliance agent using a form of the ReAct (reasoning and acting) agent framework. Beyond using large language models from Amazon Bedrock for reasoning, the agentic aspect dynamically gathers relevant signals through tool calls. Stripe chose this framework specifically to solve the problem of a near-infinite number of signals that may or may not be relevant for a given subject. Agents determine which signals are relevant and propose follow-ups until they are sufficiently confident to provide a final answer.
The ReAct cycle operates through an iterative thought-action-observation pattern. In each cycle, when a tool is requested in the Thought block, the agent framework stops LLM execution and programmatically runs that tool. It then forces that output as an observation back to the agent before allowing it to continue. This injection pattern implements what Stripe describes as a closed-loop control mechanism that serves several critical functions. It grounds agent reasoning in actual data by mandating that every tool output must be processed as an observation, preventing the agent from hallucinating or fabricating tool results. It maintains context coherence by forcing the agent to explicitly acknowledge and reason about each piece of retrieved information before proceeding. It prevents reasoning drift by acting as a checkpoint that helps verify the agent’s thought process remains anchored to factual tool outputs rather than speculative reasoning. Finally, it supports auditability by creating an explicit trace of tool invocation to observation to reasoning that can be logged for compliance review.
This approach is analogous to a feedback control system in engineering, where the agent cannot proceed to the next action without first processing the feedback (observation) from its previous action. This prevents open-loop behavior that could lead to hallucinations or off-track reasoning, which is particularly critical in financial compliance contexts where accuracy and accountability are paramount.
A significant finding from Stripe’s implementation is that traditional ML infrastructure is fundamentally unsuited for agentic workloads. Initially, Stripe attempted to fit an agent into a traditional ML inference engine, but this approach was quickly rejected for several reasons. First, the compute profiles differ dramatically: traditional ML is compute-bound, requiring expensive hardware such as GPUs, fast multi-threaded CPUs, or large memory allocations, whereas agentic applications are mostly network-bound, waiting on foundation models to finish or tool calls to run. Second, latency patterns are incompatible: an agent can take an indeterminate amount of time to finish depending on how many rounds of tool calls it needs, with a long agent query or database tool call potentially causing a thread to sit idle for minutes, compared to an XGBoost model that would finish in milliseconds. Third, the API requirements differ: traditional ML tends to output basic types like floats and Booleans, while agents need more flexibility in their schema to annotate results, and some agents need to maintain stateful conversation states.
As a result, Stripe built a dedicated agent service that initially resembled a stateless, synchronous inference endpoint but has evolved to handle stateful, multi-turn conversational agents. The service has grown from a few agents at launch to well over 100 agents in less than a year, demonstrating the scalability and generalizability of the infrastructure. This agent service sits at the center of the architecture, hosting agent logic and facilitating execution, supported by Stripe’s LLM Proxy service and connected to internal signals through available agent tools.
Rather than calling Amazon Bedrock directly, Stripe’s ReAct agents access foundation models through an LLM Proxy microservice, which has become the standard method for LLM access across the organization. This architectural pattern provides several critical capabilities for production LLMOps. First, it addresses noisy neighbor problems: with many teams using LLMs for various applications, the LLM Proxy provides safeguards preventing other teams from monopolizing LLM bandwidth for a particular model, preventing resource contention. Second, it provides a unified API across many models, simplifying how capabilities like prompt caching or tool calling are specified across foundation models from Amazon and other leading AI companies. Changing models requires only changing the model type as an argument rather than each use case managing many different clients.
The proxy also enables model fallbacks, providing the ability to automatically specify default models in case of resource constraints or outright failure, which improves reliability for production systems. Additionally, it centralizes monitoring: by requiring authentication, the service can track model usage to help forecast future resource demand and confirm appropriate models are being used depending on the privacy requirements of the application. This is particularly important for Stripe as a payment processor that must be extremely careful around privacy and security.
A significant challenge with the ReAct approach is that when a task requires many turns and observations, the prompt can become very long in later turns, particularly with verbose observations. Stripe addresses this through two complementary mechanisms. The sub-task decomposition limits the scope of each question to keep the number of turns smaller, bounding the context window requirements. More importantly, prompt caching dramatically reduces costs. With prompt caching, users only pay for the new observations and thoughts appended to previous messages at each turn rather than reprocessing the entire conversation history. Amazon Bedrock provides this capability, and Stripe’s architecture leverages it extensively. Token caching reportedly reduces costs by 60 percent by reusing common prompt prefixes across agent turns, transforming what could be a prohibitively expensive operation into something economically viable at scale.
Cost instrumentation is built into the infrastructure to track token usage per agent invocation, helping teams forecast spend as workloads scale and identify optimization opportunities before they impact budgets. This financial observability is critical for production LLMOps where cost can spiral without proper monitoring and control mechanisms.
Stripe’s use of Amazon Bedrock within its LLM Proxy provides several advantages for production LLMOps. First, it offers standardized privacy and security: as a payment processor, Stripe must be extra careful around privacy and security, and Amazon Bedrock helps verify that foundation models from Amazon and leading AI companies fit within existing security and privacy constraints without additional review overhead for each model. Second, it provides a feature-rich platform: beyond prompt caching, Amazon Bedrock supports fine-tuning and serving custom models, which Stripe expects to focus on in the coming year. Third, it offers a unified API across many models: integration is straightforward because models fall within the same API, and changing models requires only using a different model name.
Looking ahead, Stripe is exploring how to leverage Amazon Bedrock’s customization capabilities. Currently, the system uses Retrieval Augmented Generation (RAG) for dynamic knowledge injection through tool calls, giving agents access to real-time compliance data. Stripe is considering using fine-tuning capabilities to adapt model behavior specifically for financial compliance tasks, which would help lock in model quality and reduce re-evaluation overhead as models evolve. Additionally, Amazon Bedrock provides continued pre-training options for incorporating domain-specific knowledge, which could help build more specialized compliance expertise into agent reasoning. The model versioning and 6-month deprecation notice window in Amazon Bedrock helps plan these customization efforts strategically, allowing model upgrades only when they meaningfully improve investigative capabilities.
Even though Stripe ultimately uses human reviewers to make judgments and decisions, the system must still verify it stands up to regulatory scrutiny. To address this, Stripe implemented comprehensive logging so the entire agent log is retrievable for each run historically. Every agent action, decision, and rationale is documented, creating an immutable audit trail. This approach balances the efficiency gains of agentic AI with the accountability requirements of financial compliance, where regulators need to be able to trace how conclusions were reached and what information informed decisions.
The audit trail captures not just final decisions but the full ReAct cycle of thoughts, tool calls, and observations, providing transparency into the agent’s reasoning process. This is critical for identifying potential issues, debugging unexpected behavior, and demonstrating to regulators that the system operates within acceptable parameters.
A particularly important aspect of Stripe’s LLMOps approach is its commitment to testing agentic investigation components against human quality standards. The team validates with actual humans before allowing components to inform reviewers in production. This isn’t just initial testing but ongoing quality assurance, with the team exploring ways to use LLMs themselves to quickly judge and eliminate subpar approaches. This meta-level application of LLMs for quality control represents an interesting pattern where AI is used to evaluate AI, though ultimate validation remains with human experts.
The system is explicitly designed with the principle that agents assist but expert reviewers maintain final decision authority. Agents are constrained with rails to bound context and ensure they operate within well-tested domains. This human-in-the-loop design is not just about risk mitigation but about leveraging the complementary strengths of AI and human experts: agents excel at information gathering and preliminary analysis across vast datasets, while humans excel at judgment, contextual understanding, and accountability.
The production deployment achieved a 26 percent reduction in median review handling time while maintaining over 96 percent helpfulness ratings from reviewers. Critically, human reviewers remain in control of decisions, with full audit trails meeting regulatory examination standards. This represents early progress, as Stripe initially focused on questions that can be answered before the review even starts. Remaining questions likely require upstream context known and validated during the review, suggesting room for further improvement through more complex, multi-step investigations that orchestrate real-time answers as context during the review.
The efficiency gains translate directly to Stripe’s ability to scale compliance operations proportionally with business growth without corresponding headcount increases. Human reviewers can focus their time on tougher problems or new investigation opportunities, leading to an improved overall compliance program. The system also contributes to identifying 95 percent of card-testing attacks in real time and reducing unnecessary customer friction by 20 percent, demonstrating impact beyond just internal efficiency.
Through the process of building and deploying this production agentic AI system, Stripe distilled several insights that can inform similar implementations. First, keeping agent tasks bite-sized is essential: tasks should fit within working memory, and quality should be tested incrementally rather than diving straight into full automation. This task decomposition approach proved critical to making the problem tractable.
Second, orchestration matters: async workflow architecture with DAG support is essential for complex agent interactions while maintaining auditability and human oversight at scale. The ability to express dependencies between sub-tasks and pipe validated answers as context for subsequent questions creates a powerful framework for complex investigations.
Third, dedicated infrastructure is necessary: agents have fundamentally different resource profiles than traditional ML models. Traditional inference systems are compute-bound and optimized for millisecond responses on expensive GPU hardware, while agents are network-bound, spending minutes waiting on LLM calls and tool executions with unpredictable latency patterns. A dedicated agent service handles these long-running, stateful interactions through async execution patterns, allowing threads to efficiently manage multiple concurrent agent sessions without blocking on external calls.
Fourth, keeping humans in control is non-negotiable for high-stakes applications: agents assist, but expert reviewers maintain final decision authority. Constraining agents with rails to bound context ensures they operate within domains where quality has been validated and prevents them from venturing into areas where they may produce unreliable results.
While the case study presents impressive results, it’s important to maintain a balanced perspective. The 26 percent reduction in review handling time, while significant, represents early progress focused on questions that can be answered before reviews even start. The harder questions requiring upstream context represent a larger challenge, and it remains to be seen whether similar gains can be achieved as the system tackles more complex, context-dependent investigations.
The over 96 percent helpfulness ratings are self-reported from reviewers using the system, which may be subject to various biases. There’s limited discussion of failure modes, edge cases, or situations where the agents provided misleading information. Given that this is content from an AWS blog post about a customer using AWS services, there’s inherent promotional bias that may downplay challenges or limitations.
The claim that the system can “identify 95% of card-testing attacks in real time” is mentioned but not elaborated on, and it’s unclear whether this is directly attributable to the agentic system described or represents broader Stripe capabilities. Similarly, the “20% reduction in unnecessary customer friction” lacks detail about measurement methodology.
The infrastructure investment required to build a dedicated agent service, LLM proxy, and comprehensive monitoring is substantial and may not be feasible for smaller organizations. The case study reflects the resources and engineering talent available to a company at Stripe’s scale, which processes trillions of dollars annually.
Despite these caveats, Stripe’s implementation demonstrates several important patterns for production LLMOps. The distinction between agentic and traditional ML infrastructure highlights how organizations need to rethink their serving architecture as they deploy more sophisticated AI systems. The emphasis on human oversight and auditability shows how to responsibly deploy AI in regulated industries. The task decomposition and orchestration approach provides a practical framework for making complex agentic systems tractable and testable.
The LLM proxy pattern is particularly valuable as a way to standardize access, manage costs, handle fallbacks, and provide visibility across multiple teams and use cases. This kind of centralized infrastructure becomes increasingly important as LLM usage proliferates across an organization, preventing the chaos of each team independently managing relationships with model providers and implementing their own monitoring.
The commitment to testing against human quality standards before production deployment, combined with ongoing quality evaluation, represents best practices for responsible AI deployment. The exploration of using LLMs to evaluate LLM outputs suggests interesting possibilities for scaling quality assurance, though human validation remains the ultimate arbiter.
Overall, Stripe’s case study represents a sophisticated, production-grade implementation of agentic AI that balances efficiency gains with accountability, auditability, and human oversight. The architectural patterns and lessons learned provide valuable guidance for organizations considering similar deployments in high-stakes domains where errors can have significant consequences.
Stripe, processing approximately 1.3% of global GDP, has evolved from traditional ML-based fraud detection to deploying transformer-based foundation models for payments that process every transaction in under 100ms. The company built a domain-specific foundation model treating charges as tokens and behavior sequences as context windows, ingesting tens of billions of transactions to power fraud detection, improving card-testing detection from 59% to 97% accuracy for large merchants. Stripe also launched the Agentic Commerce Protocol (ACP) jointly with OpenAI to standardize how agents discover and purchase from merchant catalogs, complemented by internal AI adoption reaching 8,500 employees daily using LLM tools, with 65-70% of engineers using AI coding assistants and achieving significant productivity gains like reducing payment method integrations from 2 months to 2 weeks.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Coinbase, a cryptocurrency exchange serving millions of users across 100+ countries, faced challenges scaling customer support amid volatile market conditions, managing complex compliance investigations, and improving developer productivity. They built a comprehensive Gen AI platform integrating multiple LLMs through standardized interfaces (OpenAI API, Model Context Protocol) on AWS Bedrock to address these challenges. Their solution includes AI-powered chatbots handling 65% of customer contacts automatically (saving ~5 million employee hours annually), compliance investigation tools that synthesize data from multiple sources to accelerate case resolution, and developer productivity tools where 40% of daily code is now AI-generated or influenced. The implementation uses a multi-layered agentic architecture with RAG, guardrails, memory systems, and human-in-the-loop workflows, resulting in significant cost savings, faster resolution times, and improved quality across all three domains.
