Google Cloud and LangChain addressed the infrastructure challenges of running AI agents at scale by developing agent sandbox capabilities on Google Kubernetes Engine (GKE). The problem centered on providing isolated, low-latency runtime environments for autonomous agents that could execute untrusted code safely while maintaining high throughput and resource efficiency. The solution leveraged GKE's agent sandbox technology with gVisor kernel isolation, subsecond sandbox provisioning (300 sandboxes per second), and snapshot/restore capabilities to suspend idle agents and clone running states. Results include 89% reduction in time-to-first-token for model serving, approximately 90% reduction in compute consumption through suspend/resume cycles, sub-500ms sandbox allocation latency, and the ability to run hundreds to thousands of concurrent isolated agent runtimes with enterprise-grade security and observability.
This case study presents a comprehensive look at building production infrastructure for AI agents at scale, featuring collaboration between Google Cloud’s GKE team and LangChain. The session focused on the technical architecture required to run autonomous agents safely and efficiently in production environments, particularly addressing the challenges of executing untrusted code, managing resource consumption, and achieving the performance characteristics necessary for modern agent workloads.
LangChain’s evolution provides context for understanding the infrastructure requirements. Starting with the LangChain framework in October 2022, the company progressed through observability tooling in mid-2023, deterministic workflow support with LangGraph, deployment runtimes, and most recently deep agents for long-running autonomous use cases. Each evolution demanded more sophisticated infrastructure capabilities, culminating in the need for isolated, ephemeral, yet persistent sandbox environments.
The presenters outlined how agent capabilities have evolved rapidly from basic function calling to Model Context Protocol integration, computer use, coding agents, and now persistent agents that run continuously. Modern agents like those demonstrated with tools such as Cloud Code and open-source alternatives require more than just intelligent models. They need comprehensive compute environments that can handle orchestration, code execution, terminal and browser access, tool coordination, context management across multiple agents, and serving the results back to users.
The concept of an “agent harness” emerged as the term describing this complete compute environment. When combined with the intelligence layer of the LLM itself, the harness provides the full infrastructure stack needed for production agent systems. The critical insight is that agents executing code represent potentially untrusted workloads that require absolute kernel-level isolation, not just container boundaries.
The technical requirements vary by agent type but generally include several critical capabilities. For code execution tools, computer use agents, and terminal agents that perform specific tasks with human-in-the-loop interaction, the system needs kernel-level isolation to protect against zero-day vulnerabilities, network isolation for security boundaries, and very low latency provisioning in the 500 milliseconds or less range. For persistent agents running continuously, additional requirements include snapshotting capabilities to maintain agent state including memory and filesystem contents, the ability to fork processes for parallel execution, and rapid suspend/resume functionality to optimize resource utilization.
Scale represents another fundamental requirement. Production agent systems aren’t deploying tens of agents but hundreds, thousands, or even millions of concurrent agent instances. This scale demand drove many of the architectural decisions around the infrastructure approach.
The presenters positioned three broad infrastructure approaches on a spectrum. At one end sits fully managed agent platforms like Gemini Enterprise that provide turnkey solutions with built-in isolation, low latency, and snapshotting but with less customization. At the other end lies DIY compute where organizations provision and manage their own VM instances, gaining maximum control but shouldering all management burden and potentially missing some advanced capabilities.
Kubernetes, and specifically GKE, occupies the middle ground that the teams identified as optimal for most agent builders. It provides a managed control plane while remaining highly customizable, supports bringing your own compute, and extends through standard APIs. However, standard Kubernetes implementations have gaps for agent workloads, particularly around low latency provisioning (typically 1-3 seconds or more) and snapshotting capabilities.
Google Cloud’s agent sandbox for GKE addresses these gaps through several technical innovations. Built on open-source Kubernetes SIG work, the agent sandbox provides high-throughput scheduling capable of 300 sandboxes per second with subsecond latency. The architecture integrates deeply with Google’s Axion processors for price-performance optimization and connects with the full GCP stack for identity and access management, network policies, governance, and observability down to the compute layer.
The low-latency provisioning works through a warm pool mechanism. Rather than spinning up compute on-demand when a user or agent requests a sandbox, the system maintains a pre-warmed pool of ready sandbox instances. When a request arrives, a sandbox is immediately allocated from the pool with the pool backfilled asynchronously. This architectural pattern enables the subsecond latency critical for agent interactions where users expect immediate responses.
For resource efficiency with bursty agent workloads, the suspend/resume capability proves critical. Agents often have significant idle periods, particularly long-running coding agents waiting for human input. The system can snapshot the exact state of an idle sandbox including memory and filesystem, store that state in Google Cloud Storage, and suspend the compute resources. When the agent becomes active again, the snapshot is rehydrated rapidly, restoring the agent to its previous state. This capability was demonstrated to reduce compute consumption by approximately 40% in typical workloads and up to 90% in some scenarios.
The isolation layer relies heavily on gVisor, Google’s specialized sandboxing technology used extensively within Google’s own infrastructure including by DeepMind for running Gemini-generated code. Unlike traditional virtualization or microVMs, gVisor takes a unique approach by reimplementing the entire Linux kernel interface in userspace. This includes memory management, filesystems, network stack, and process management.
The security model creates defense-in-depth through two distinct layers. An attacker would need to compromise both the gVisor userspace kernel and then escape the Linux kernel isolation primitives that protect the host system. The syscall interception redirects to the reimplemented kernel rather than filtering, providing comprehensive coverage. Higher-privilege operations use a separate companion process called the gofer to further isolate risky operations.
For agent workloads, gVisor provides several advantages over alternatives like full VMs or microVMs. It’s fully container-native with support for burstable CPU and memory, enabling higher density on shared hosts. The overhead characteristics differ from nested virtualization as well. While nested virtualization might add 8-10% performance overhead, gVisor CPU and memory operations inside the sandbox experience essentially no overhead with native performance. The syscall interception does add latency, but for most applications performing IO and computation, this proves negligible. Importantly, gVisor also supports GPU acceleration with very low overhead for agents requiring accelerators.
The snapshotting functionality represents a key differentiator enabled by gVisor’s architecture. Because all processes and child containers within a sandbox live in the same managed unit with a reimplemented kernel, the system has complete visibility into all state, dependencies, open connections, and memory contents. This enables reliable snapshot, restore, and clone operations even for complex application states.
The clone capability in particular enables powerful optimization patterns. For model inference serving, snapshotting a server just as it becomes ready to serve and then cloning that state for scale-out dramatically reduces time-to-first-token. The presenters cited 89% reduction in time-to-first-token as a concrete result, with this metric improving continuously from 50% to 70% to the current 89% as the team optimizes the implementation. The same pattern applies to slow-loading applications like Java services where bootstrap time can be amortized across many cloned instances.
For agent workflows, cloning enables patterns like the demonstrated multi-agent collaboration where an architect agent develops initial context, that state is cloned, and then multiple specialist agents work in parallel from the same baseline but with different focus areas for optimization, coding, security analysis, and review. Each specialist inherits the full context without redundant computation.
Snapshots are exposed through policy-based declarative Kubernetes resources. Workloads can trigger snapshots on-demand via the Kubernetes API or integrate triggering directly into application code for precise timing control. The system manages the complete lifecycle, storing snapshots in Google Cloud Storage automatically. Snapshots capture CPU and memory state, GPU memory, the complete filesystem, and all open connections so that restored sandboxes resume listening on ports and maintaining network state.
LangChain provides a simple Python API for consuming snapshots within agent code, making the capability accessible without deep Kubernetes expertise. The integration also works through Kubernetes Custom Resource Definitions for teams preferring declarative infrastructure management.
LangChain’s Eric Johansen provided the customer perspective on using this infrastructure in production. LangChain’s entire SaaS platform runs on Google Cloud Platform with GKE as the underlying infrastructure, specifically chosen for the built-in sandbox runtime capabilities. The company’s product evolution created increasing infrastructure demands, from the initial framework through observability tooling, deterministic workflows with LangGraph, deployment runtimes, and finally deep agents for complex long-running autonomous use cases.
The security and isolation requirements proved particularly critical. LangChain received immediate feedback after releases of tools like open-source coding assistants that customers couldn’t adopt them without proper security and governance layers. The gVisor-based sandboxing provided the kernel isolation and network policy controls necessary for enterprise adoption. Every sandbox runs in its own lightweight microVM with full kernel isolation beyond just container boundaries. Kubernetes network policies lock down traffic to prevent sandboxes from reaching internal services, and users can bring any Docker image with LangSmith handling the orchestration.
The managed aspects of GKE proved equally important. LangChain doesn’t manage any scaling manually. The platform automatically scales for varying throughput from multiple concurrent users and resource-intensive agents. Workload identity integration, GCP ecosystem benefits, and the ability to handle customers generating anywhere from millions to billions of traces demonstrate the production-readiness of the infrastructure. The automatic scaling handles both the extreme variance between customers and the bursty nature of individual agent workloads.
LangChain’s approach emphasizes reducing the barrier between prototyping and production deployment. The LangGraph CLI tools enable developers to quickly initialize projects with example code tailored to specific use cases. LangGraph Dev provides a graphical representation of agent workflows where developers can visualize nodes, test different runtime configurations, and manage state at different execution points for debugging specific issues.
The deployment story centers on simplicity through the LangGraph Deploy command. As teams iterate on agent development with multiple contributors, a single command deploys the agent into LangSmith’s infrastructure backed by GCP, immediately providing production runtime access with eight endpoints, MCP integration, and other management features out of the box.
For deep agents specifically, the framework provides built-in features that would otherwise require significant custom development. Native context management handles per-user context rather than forcing all context into system prompts. Long-term and short-term memory capabilities integrate into the framework. Dynamic context loading about users and their use cases scales to thousands of users without manual memory management. The model-agnostic design allows teams to swap models as capabilities improve and evaluate agent output across different model choices. The deep agent CLI provides a coding assistant with native understanding of the framework’s features similar to other AI coding tools.
The presentations included two substantial demonstrations of the infrastructure capabilities. The first showed a playful but technically accurate simulation with hundreds of concurrent agent instances performing different operations. The demo scaled from 4 agents to 200 agents rapidly, with clear visibility into creation, suspension, and resume cycles. The warm pool mechanism enabled the fast scale-up, while the snapshotting capability allowed agents to pause and resume as they transitioned between active and idle states.
The second demonstration presented a more realistic scenario with a multi-agent coding workflow. An architect agent developed initial design context, which was then cloned across four specialist agents for optimization, implementation, security analysis, and code review. Each specialist worked in parallel from the same baseline context but with different focus areas. As agents completed their work, they turned gray in the visualization to indicate suspension after snapshotting their state. IP addresses changed between executions, demonstrating that sandboxes were scheduled across the cluster, suspended, and resumed on different nodes seamlessly.
The demo then scaled this pattern significantly higher while showing real-time performance metrics. Suspend times around 100 milliseconds and restore times in the 200-280 millisecond range enabled rapid cycling between active and idle states. The compute efficiency gains from suspending idle agents rather than letting them consume resources showed the approximately 90% reduction in compute consumption mentioned earlier. The demonstration ran actual isolated sandbox instances using gVisor, not simulated workloads, proving the production viability of the approach.
The presenters highlighted Lovable as a customer building on the sandboxing stack at scale. Lovable specifically called out GKE’s sandboxing capabilities for enabling reliable scaling to hundreds of secure sandboxes per second, which proved critical for seamlessly empowering their users even during massive unpredictable demand spikes. This validates that the infrastructure handles both the steady-state scale requirements and the elasticity needs of real-world agent platforms.
A significant aspect of the case study involves the open-source nature of much of this work. The agent sandbox builds on Kubernetes SIG work that Google has contributed to for several months alongside industry partners. This community-driven development ensures the solutions aren’t proprietary and can benefit the broader ecosystem. LangChain similarly emphasizes open-source development with public repositories for their frameworks and tools.
Google provides hands-on labs for building with agent sandbox on GKE, comprehensive documentation covering technical details, and encourages community contribution to the Kubernetes SIG efforts. This approach balances Google’s commercial interests in GKE adoption with genuine contributions to open infrastructure for the AI agent ecosystem.
The case study presents a compelling technical solution but deserves balanced assessment. The infrastructure clearly addresses real production challenges around isolation, latency, resource efficiency, and scale that agent builders face. The performance metrics around sandbox provisioning speed, suspend/resume cycles, and time-to-first-token improvements appear substantial and were demonstrated with actual workloads rather than just theoretical claims.
However, the presentation comes from Google Cloud and a partner heavily invested in the GCP ecosystem, so some commercial motivation exists. The comparisons to alternative approaches like DIY VM management or microVMs could be seen as somewhat simplified, though the core trade-offs identified seem technically sound. The Kubernetes middle-ground positioning serves Google’s interests in GKE adoption while also reflecting genuine advantages in control plane management and API extensibility.
The gVisor technology represents real innovation with production validation through Google’s internal use, but organizations should evaluate whether the specific performance characteristics and security model fit their requirements versus alternatives like Firecracker microVMs or Kata Containers. The syscall interception approach does add latency in some scenarios even if negligible for many workloads.
The resource efficiency claims deserve scrutiny in specific contexts. The 40-90% compute reduction depends heavily on agent workload patterns and how bursty or idle they are in practice. Teams with agents that remain continuously active may see less benefit from suspend/resume capabilities, though the snapshot/clone functionality still provides value for scaling.
The integration with GCP services like Cloud Storage for snapshots, workload identity, and observability creates some ecosystem lock-in even though the underlying technologies are open source. Teams committed to multi-cloud strategies or other Kubernetes platforms should evaluate portability implications.
Overall, the case study demonstrates sophisticated LLMOps infrastructure that addresses genuine production challenges with measurable results. The open-source foundation, production validation through customers like Lovable and LangChain’s own platform, and concrete performance demonstrations lend credibility. Organizations building agent platforms at scale should seriously evaluate this approach while considering their specific requirements, existing infrastructure investments, and cloud strategy.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Notion, a knowledge work platform serving enterprise customers, spent multiple years (2022-2026) iterating through four to five complete rebuilds of their agent infrastructure before shipping Custom Agents to production. The core problem was enabling users to automate complex workflows across their workspaces while maintaining enterprise-grade reliability, security, and cost efficiency. Their solution involved building a sophisticated agent harness with progressive tool disclosure, SQL-like database abstractions, markdown-based interfaces optimized for LLM consumption, and a comprehensive evaluation framework. The result was a production system handling over 100 tools, serving majority-agent traffic for search, and enabling workflows like automated bug triaging, email processing, and meeting notes capture that fundamentally changed how their company and customers operate.
This case study presents Dapr, a CNCF graduated project, and its application to production AI agent systems through the Dapr Agents framework. The core problem addressed is the unreliability of current agent frameworks when running at scale in production environments, particularly the challenge of state loss during failures that forces expensive re-execution of long-running agentic workflows. Dapr Agents provides a durable agent framework with built-in workflow orchestration, automatic failure detection and recovery, exactly-once execution guarantees, and support for over 30 different state stores. The solution was demonstrated through live examples showing agents automatically resuming from their exact point of failure without manual intervention, multi-agent collaboration using pub/sub mechanisms, and complete observability through OpenTelemetry integration. Contributed by Nvidia to the Dapr project and reaching 1.0 stability in 2026, the framework addresses critical production gaps in existing agent frameworks like LangChain and LangGraph.
