PayPal's SRE team faces operational challenges at planetary scale, managing 450 million users, 3,000 microservices, and 2 billion daily API interactions with zero margin for error. The company partnered with Google to build an autonomous agentic SRE ecosystem that transforms traditional reactive incident response into proactive, AI-driven operations. The solution employs specialized AI agents that collaborate in a mesh architecture, handling everything from architecture validation and deployment rollouts to incident detection, troubleshooting, and remediation. The system operates through three layers: a data lake with MCP servers, real-time telemetry processing, and intelligent action orchestration via a supervisor agent. Early results suggest the potential to reduce failure detection from 10% to 1% of rollout completion, dramatically improving mean time to detection and mitigation while enabling autonomous incident lifecycle management.
This case study from Google Cloud Next 2025 presents a comprehensive vision and early implementation of agentic AI systems for Site Reliability Engineering, with PayPal serving as the primary production use case. PayPal operates at planetary scale, managing 450 million users, processing financial transactions worth 5 million in revenue per minute, maintaining 3,000 microservices, handling 2 billion daily API interactions, and executing 10.5 million changes annually. The stakes in this environment are exceptionally high, with zero margin for error.
The fundamental challenge that motivated this transformation is structural: while developers focus on specific microservices, SREs are expected to manage the entire stack end-to-end. This creates three critical problems: cognitive overload during incidents where engineers must synthesize information from hundreds of fragmented dashboards while applying tribal knowledge; human inconsistency where incident response quality varies based on who happens to be on call; and reactive-by-default operations where failures have already cascaded by the time connections are made.
A concrete example from PayPal’s Cyber 5 period illustrates these challenges. During this peak traffic event when normal traffic increases three to four times, a load test discovered inconsistent memory saturation on critical nodes that was invisible during normal operations. Debugging required multiple hours with multiple teams working together, and had this issue reached production during peak traffic, it would have caused cascading failures severely impacting merchants and customers.
The core architectural innovation presented is moving from monolithic dashboards and reactive operations to what is described as an “agentic mesh” - a swarm of highly specialized AI agents that collaborate to provide end-to-end reliability. This represents a fundamental shift in how SRE operations are conceptualized and executed.
The ecosystem consists of specialized engineering teams in the form of AI agents, where each agent is an expert in a specific domain. An orchestrator coordinates end-to-end flows, while agents and sub-agents solve smaller problems within the overall workflow. Critically, these agents do not operate in silos - they continuously communicate and share context, creating a cohesive unit operating 24/7 that watches, reacts, and closes loops of understanding and remediation.
Specific agents described in the architecture include: an architect agent ensuring designs are reliable by design and removing single points of failure; a design and configuration agent tracking experiments behind feature flags; a rollout agent monitoring deployment progression; a health agent understanding signals for workload health; an incident investigation agent; and a postmortem agent that tracks action items. Each agent contributes its specialized capability to create more complex end-to-end flows.
The case study presents a thoughtful framework for gradually introducing autonomous agents, recognizing that comfort levels with agent decision-making will evolve over time. Two primary patterns are discussed:
Human-in-the-loop mode represents the initial pattern where agents do the heavy lifting but pause for human review and consent before executing actions like applying remediations or generating reports. This allows verification of agent decisions while still benefiting from their analytical capabilities.
Autonomous mode represents increased autonomy where agents execute tasks without human involvement. The implementation strategy acknowledges that autonomy levels may vary by environment - organizations might grant more autonomy in development environments while requiring approvals for production, though this is expected to evolve toward full production autonomy over time.
The implementation leverages five identified engineering patterns for agent orchestration, though the presenters note many more patterns are likely to emerge. These patterns enable both sequential operations (rollout, execute task, continue rollout) and parallel operations for complex workflows. Hierarchical task decomposition allows maintenance operations to be broken down into manageable sub-tasks handled by specialized sub-agents. The pattern-based approach provides flexibility for organizations to compose operations in ways that match their specific operational needs and constraints.
To illustrate concrete agent capabilities, the presentation walks through a hypothetical shopping cart application running on Google Cloud infrastructure with a Cloud Run frontend, GKE backend, Cloud Spanner database, and third-party payment service integration. This example demonstrates the breadth of specialized agents throughout the software lifecycle:
The architect agent addresses a critical challenge in the AI coding era - the proliferation of AI-generated code where quality and reliability are unknown. This agent reviews shopping cart architecture to detect and remove single points of failure, identify multi-threading issues, and ensure proper separation of data plane, control plane, and management plane concerns. Alongside it, the reliable design and config agent reviews deployment configurations and Terraform to ensure exponential backoff is configured, throttling is implemented, and best practices are encoded. For a service requiring 24/7 availability, it ensures multi-region deployment across continents with appropriate resilience patterns.
The rollout agent represents a particularly sophisticated capability. Given the statistic that over 70% of outages occur during rollouts of binaries or configurations, this agent’s correlation capabilities become critical. It understands the service dependency graph, selects appropriate rollout policies based on those dependencies, and determines whether to rollout zone-by-zone, region-by-region, follow-the-sun, or use progressive rollout strategies. The key innovation is in detection: while humans currently detect rollout problems at approximately 10% completion, the expectation is that LLMs can detect issues at just 1% completion - a 10x improvement that dramatically reduces blast radius and improves reliability.
The experiment agent manages feature rollouts like product recommendations. Again, the detection capability is transformative: while current human-driven processes detect experiment failures at around 10% of the affected user population, agent-based systems are expected to detect failures at 1% of 1% of users - identifying issues affecting just 0.01% of the experiment population and automatically rolling back. This enables much safer experimentation at scale.
The traffic agent addresses the challenge of differentiating organic traffic spikes from denial-of-service attacks. The shopping cart example references the infamous Taylor Swift concert ticket sale where systems crashed under load, raising questions about whether traffic was legitimate demand or an attack. The agent can observe anomalies as they build up rather than waiting for threshold-based alerts, separate valid traffic from malicious traffic, trigger auto-scaling, and even orchestrate capacity management by purchasing additional capacity if allowed or coordinating with launch agents to pause other launches temporarily to preserve capacity for critical services.
The observability agent tackles the complex task of properly instrumenting applications. It identifies appropriate locations in code for metrics, logs, and traces - emitting what is needed without over-instrumenting. It ensures errors are logged in actionable format (IAP format), installs cloud observability sidecars for metric collection, and inspects deployments to ensure observability services are enabled. This automation addresses a traditionally time-consuming manual process.
The SLI agent demonstrates impressive analytical capabilities. When given the shopping cart architecture and asked to define Service Level Indicators, Gemini generated comprehensive SLIs including: availability SLIs for add, update, remove, and initiate checkout operations; latency SLIs specifying add-to-cart should complete in under 300ms and view-cart in under 200ms; and correctness SLIs for data integrity. Notably, the LLM even provided guidance on SLI design best practices, recommending separation of write and read paths. The agent can design SLIs in minutes rather than the months typically required, then orchestrate code changes to emit the necessary metrics for SLO monitoring.
The maintenance agent handles schema changes and other maintenance operations safely. For a database schema upgrade, it determines service dependencies, traces back to traffic sources (internet, VPC, or other services), and administratively moves traffic away to different regions. It executes a workflow with pre-event customization, performs maintenance when traffic is drained, runs post-operations for service warm-up, and then undrains traffic - all while maintaining continuous service availability.
The health agent operates across multiple layers of the stack, probing climate and geopolitical events (earthquakes, floods, wars, data center unavailability), physical infrastructure (cooling, power, submarine cable failures), data center infrastructure and services, cloud services, and application-level issues. While probing infrastructure itself doesn’t require agents, the critical capabilities are correlation and noise removal, plus anomaly detection that identifies issues before alerts fire. This enables near-zero mean time to detection and mitigation.
When issues are detected, the troubleshooting process leverages parallel investigation - a capability impossible at human scale. Multiple sub-agents simultaneously evaluate different hypotheses: one examines network conditions, another reviews recently deployed code, another investigates database transaction performance. This parallel investigation eliminates tunnel vision where responders might pursue a single incorrect hypothesis, and dramatically reduces mean time to resolution.
The incident agent manages the complete incident lifecycle end-to-end. Beyond diagnosis and launching parallel investigations, it analyzes user and customer impact, drafts reports explaining what happened and who was affected, targets communications to impacted users, creates postmortems, generates action items to prevent recurrence, and tracks remediation fixes to completion. This represents a comprehensive closure of the incident loop from detection through communication to long-term prevention.
PayPal has moved beyond conceptual design to operational deployment with a live SRE ecosystem built on three architectural layers. The data layer consists of MCP servers that expose data from microservices and runtime services flowing into an SRE data lake. These MCP servers provide the interfaces through which workflows access operational data.
The signal processing layer handles real-time telemetry that gets enriched by AI agents. Rather than simply ingesting raw telemetry, this layer adds intelligence and context before passing information to higher-level agents.
The intelligent action layer features the supervisor agent that orchestrates the entire incident lifecycle autonomously. This supervisor coordinates the specialized agents, manages workflows, and ensures appropriate responses to operational events. The entire system is built on Google’s AI stack and Cloud Assist platform.
PayPal’s implementation demonstrates the transition from monolithic operations to what they characterize as a flexible and intelligent mesh where each agent is a first-class resource. This keeps PayPal’s global economy moving while enabling the SRE team to scale beyond human limitations.
Several important LLMOps considerations emerge from this case study, though many details remain at the vision stage given this was a 2025 conference presentation discussing forward-looking roadmaps. The presenters explicitly disclaim that the content represents current expectations subject to change, particularly in this rapidly evolving space.
The architecture demonstrates sophisticated multi-agent orchestration where context sharing between agents is critical. Each agent must understand not only its own domain but also receive context from upstream agents and provide context to downstream agents. This requires robust interfaces, well-defined contracts, and careful state management.
The gradual rollout of autonomy levels represents a pragmatic LLMOps approach, acknowledging that organizations need to build confidence in agent capabilities before granting full autonomy, particularly in production environments where financial transactions are at stake. The differentiation between development and production autonomy levels provides a practical path for organizations to gain experience with agent capabilities in lower-risk environments.
The emphasis on detection thresholds improving from 10% to 1% represents a quantitative claim about LLM capabilities in operational contexts. While ambitious, these claims require validation through production deployment and measurement. The case study would benefit from concrete metrics on actual detection rates, false positive rates, and the precision-recall tradeoffs inherent in anomaly detection systems.
The pattern-based orchestration approach provides reusability and standardization benefits, allowing organizations to compose complex workflows from well-understood building blocks. This is particularly important for LLMOps at scale where ad-hoc agent interactions would quickly become unmanageable.
Integration with existing Google Cloud infrastructure (Cloud Run, GKE, Cloud Spanner) demonstrates the practical grounding of agent capabilities in real infrastructure APIs and management planes. The agents must interact with these systems through well-defined interfaces, handle API failures gracefully, and manage eventual consistency and distributed system challenges.
The observability requirements for the agents themselves represent a meta-level LLMOps challenge. How do you monitor agents that are themselves responsible for monitoring? What metrics indicate agent health versus agent effectiveness? How do you debug agent decisions when they operate autonomously? While the presentation doesn’t address these operational challenges explicitly, they are critical for production deployment.
This case study represents an ambitious vision for AI-native operations with early production validation from PayPal. Several aspects deserve balanced consideration:
The quantitative claims about 10x improvement in detection rates are compelling but require empirical validation over extended periods. Anomaly detection systems historically struggle with false positive rates, and the practical deployment challenge will be tuning these systems to maintain accuracy while minimizing alert fatigue.
The parallel investigation capability represents a genuine innovation over human-driven processes, though the coordination complexity grows with the number of parallel agents. Managing consistency when multiple agents propose conflicting hypotheses or remediations will require sophisticated orchestration logic.
The architecture’s dependency on accurate dependency graph understanding is critical but challenging. Many organizations struggle to maintain accurate service dependency information, and agent decisions based on incomplete or incorrect dependency data could cause cascading issues rather than preventing them.
The human-in-the-loop versus autonomous modes acknowledge a real tension in production operations. Organizations with regulatory requirements or risk-averse cultures may remain in human-in-the-loop mode longer than the vision suggests, potentially limiting the speed benefits of autonomous response.
The Gemini-generated SLI example is impressive but also illustrates the challenge of validating LLM outputs. While the suggested SLIs appear reasonable, determining whether 300ms is the right threshold for add-to-cart versus 200ms or 400ms requires domain expertise and business context that may not be fully captured in the architecture description provided to the LLM.
Overall, this case study represents significant innovation in applying agentic AI to production operations. PayPal’s operational scale and zero-error-tolerance environment provide credible validation that these approaches can work in demanding production contexts. The architectural patterns and agent specialization strategy provide a blueprint for other organizations pursuing AI-native operations. However, the transition from vision to full production deployment will require solving numerous LLMOps challenges around agent coordination, verification of agent decisions, handling edge cases, and maintaining reliability when the reliability system itself is AI-driven.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Two organizations operating in highly regulated industries—Sicoob, a Brazilian cooperative financial institution, and Holland Casino, a government-mandated Dutch gaming operator—share their approaches to deploying generative AI workloads while maintaining strict compliance requirements. Sicoob built a scalable infrastructure using Amazon EKS with GPU instances, leveraging open-source tools like Karpenter, KEDA, vLLM, and Open WebUI to run multiple open-source LLMs (Llama, Mistral, DeepSeek, Granite) for code generation, robotic process automation, investment advisory, and document interaction use cases, achieving cost efficiency through spot instances and auto-scaling. Holland Casino took a different path, using Anthropic's Claude models via Amazon Bedrock and developing lightweight AI agents using the Strands framework, later deploying them through Bedrock Agent Core to provide management stakeholders with self-service access to cost, security, and operational insights. Both organizations emphasized the importance of security, governance, compliance frameworks (including ISO 42001 for AI), and responsible AI practices while demonstrating that regulatory requirements need not inhibit AI adoption when proper architectural patterns and AWS services are employed.
Fidelity Investments faced the challenge of managing massive volumes of AWS health events and support case data across 2,000+ AWS accounts and 5 million resources in their multi-cloud environment. They built CENTS (Cloud Event Notification Transport Service), an event-driven data pipeline that ingests, enriches, routes, and acts on AWS health and support data at scale. Building upon this foundation, they developed and published the MAKI (Machine Augmented Key Insights) framework using Amazon Bedrock, which applies generative AI to analyze support cases and health events, identify trends, provide remediation guidance, and enable agentic workflows for vulnerability detection and automated code fixes. The solution reduced operational costs by 57%, improved stakeholder engagement through targeted notifications, and enabled proactive incident prevention by correlating patterns across their infrastructure.
