Google's Chrome DevTools team built a purpose-built version of their developer tools specifically for AI agents, addressing the challenge that coding agents were "flying blind" when generating code without being able to validate their outputs. The team discovered that agents couldn't process raw performance traces containing 50,000+ lines of JSON, so they redesigned their interfaces to provide semantic summaries in markdown instead of raw data. Through iterative development, they decomposed their initial monolithic tool into 25 specialized tools, introduced metrics for measuring "tokens per successful outcome" to balance effectiveness with efficiency, implemented error recovery mechanisms for self-healing agents, and designed security boundaries that maintain human oversight. The solution works with any MCP-capable agent harness including Gemini CLI, Claude Code, and others, enabling agents to debug web pages, analyze performance, and validate improvements autonomously.
Google’s Chrome DevTools team developed Chrome DevTools for Agents, a production deployment of browser automation and debugging capabilities specifically designed for AI agents rather than human developers. This case study is presented by Michael Hablich, Product Manager for Chrome Developer Tools at Google, and represents approximately 1.5 years of iterative development starting from early observations that coding agents were highly capable at generating code but unable to validate what they were actually producing.
The traditional Chrome DevTools is used by millions of web developers daily for debugging web pages, finding errors, auditing performance, and profiling applications. The agent-focused version maintains similar functional goals but requires fundamentally different interface design decisions based on the cognitive differences between human and agent users.
The initial approach to agent integration revealed a critical misconception. The team initially assumed that because agents are machines, they could simply be given massive amounts of raw data and would process it effectively. This assumption proved incorrect when they attempted to feed raw performance trace files to agents.
A typical performance trace file contains multiple megabytes of data, often exceeding 50,000 lines of JSON with granular details about every aspect of webpage performance. When this raw data was provided to agent harnesses, it consistently blew through context windows and pushed agents into what was referred to as “the dump zone” - a state where agents are overwhelmed with too much information to reason effectively about the task at hand.
This failure led to a fundamental rethinking of how to present information to agents. Rather than forcing agents to “read the entire book,” the team pivoted to pointing agents at “the right sentence” - providing semantic summaries and preprocessed insights rather than raw data dumps.
The performance tracing endpoints were redesigned to return markdown-formatted semantic summaries instead of raw JSON traces. These summaries provide information about typical web performance metrics like Largest Contentful Paint (LCP), Interaction to Next Paint (INP), Cumulative Layout Shift (CLS), and other Core Web Vitals in a format that agents can efficiently process and act upon.
This architectural decision represents a key LLMOps principle: interface design for agents must account for token economics and cognitive processing patterns that differ fundamentally from human users. While humans benefit from visual complexity, color, and layout to find signals in data, agents require structured, semantically meaningful text that fits within context window constraints.
A crucial insight from the project was recognizing that agents represent a distinct user segment with their own non-functional requirements. While agents and humans share the same intent and goals - such as identifying and fixing errors in web pages - they have fundamentally different cognitive bottlenecks.
For humans, visual complexity is the primary challenge, requiring careful UI/UX design with appropriate visual hierarchies and layouts. For agents, the bottlenecks involve context window limitations, token costs, and the ability to discover and correctly invoke tools from potentially large tool libraries.
The team introduced a key metric for evaluating agent interface effectiveness: tokens per successful outcome. This metric captures what they describe as the “fuel efficiency” of the interface, measuring not just token cost or number of tool calls, but specifically the cost of successful task completions.
The metric has two critical dimensions:
Effectiveness measures whether the agent completes the entire user journey and fulfills the functional intent. This is a binary yes/no assessment of task completion.
Efficiency measures token cost, number of tool calls, and duration. However, as the team emphasizes, fuel efficiency is worthless if you can’t reach your destination - hence the focus on tokens per successful outcome rather than just tokens per outcome.
An important caveat is that this metric cannot be meaningfully compared across different user journeys or task classes. In Chrome DevTools for Agents, web scraping tasks are relatively token-efficient since they involve straightforward information extraction. In contrast, debugging responsive layout issues requires much more intricate and interactive sessions with higher token consumption - but this is appropriate given the task complexity. The team tracks these metrics separately for different use cases, visualizing them as comparison bars where longer bars indicate more effective tools for particular scenarios.
The team acknowledges that measuring tokens per successful outcome is not straightforward, but maintains that even imperfect measurement enables data-informed decisions rather than purely gut-driven development choices.
Chrome DevTools for Agents addresses token consumption from three different angles:
Tool Categorization involves hiding niche tools behind command line parameters. For example, Chrome extension debugging tools are not included in the default context since most developers aren’t working on extensions. This reduces the default context window size while maintaining capability for specialized use cases.
Slim Mode pushes categorization to its limits by exposing only three core tools: select page, navigate page, and evaluate script. This minimizes context window consumption but creates an important trade-off: fewer tools mean agents may require extra turns to achieve goals or may lack the right tools entirely for certain tasks. For instance, getting network requests cannot be accomplished with just evaluate script.
Command Line Interface support enables agents to chain commands together for post-processing. Rather than having the model process all tokens, operations like extracting accessibility tree data with grep and piping results into click commands allow local post-processing, significantly reducing token consumption since the LLM doesn’t need to process intermediate data.
The initial design featured a single monolithic tool called “debug webpage” where agents could send prompts describing what they wanted to debug. While elegant from an engineering perspective, this approach failed in practice.
The team decomposed this single tool into 25 different specialized tools, each with specific functionality. However, this created a new problem: discoverability. With 25 tools available, how do agents determine which tool to use when?
The team references research indicating that 97% of MCP tool descriptions have quality smells - a critical issue since the schema functions as the UI for agents. Improving descriptions creates its own trade-off: better descriptions increase context window size, and smaller models in particular can become biased toward using tools they shouldn’t use when given overly detailed descriptions.
Despite these trade-offs, certain description practices are relatively uncontroversial and beneficial:
Define Purpose by clearly explaining the tool’s core function. For the performance start trace tool, the description states: “Used to find performance, front-end performance issues and core web vitals, LCP, INP, CLS.” The specific mention of metric acronyms enables agents to make connections like “I should use this tool if I need to improve page load.”
Usage Guidelines provide clear activation criteria that help agents understand when specific tools are appropriate.
The team frames this as an endless quest for minimum viable descriptions, acknowledging that optimal descriptions shift as models and agent harnesses continue evolving.
Beyond individual tool descriptions, Chrome DevTools for Agents provides skills - higher-level workflow patterns for more intricate operations. However, skills are not free lunch: too many skills shift the discoverability problem rather than solving it, potentially causing agents to invoke skills inappropriately while also increasing context window size.
One example is a troubleshooting skill that helps agents and humans fix setup issues with the MCP server configuration, enabling self-healing when installation problems occur.
Every time an agent encounters an error, it costs tokens for retry attempts and error analysis. The team approaches error recovery as a spectrum with multiple strategies:
Useful Error Messages that were iteratively improved across tools. For example, when unable to navigate back in page history, the error message explicitly explains that the history entry was not found and provides guidance on corrective action. This additional context enables agents to self-heal without human intervention.
Proactive Detours involve deliberately counteracting model training data when beneficial. The system proactively redirects agents toward the start performance trace tool rather than Lighthouse audit for performance profiling tasks, based on empirical understanding of which approach works better for agent workflows.
Diagnostic Playbooks like the troubleshooting skill mentioned earlier help agents diagnose and fix common configuration issues autonomously.
All of these mechanisms increase system resilience and reduce the frequency of human intervention required to unstick agents.
Chrome DevTools for Agents includes an autoconnect feature that allows humans to share their screen with agents during debugging sessions - for example, asking Claude Code to help debug a problem visible in the browser. Users requested the ability to “remember my choice” and avoid repeated permission prompts.
In traditional UX design, this would be an obvious win for reducing friction. However, in agent-based systems, the team argues that friction can be a feature rather than a bug when it comes to trust boundaries.
Drawing on the concept of the lethal trifecta, the team identifies three tiers of security posture for browsing agents:
Tier One: Local Development Environment involves human-in-the-loop scenarios where the human grants time-bound access to their default Chrome profile and existing data. In this tier, the human needs to explicitly consent each time the agent connects, maintaining oversight even though it creates friction.
Tier Two: Continuous Integration Environments are controlled but separated environments where data separation mechanisms like containers and separate Chrome profiles should be used. Remote debugging port capabilities support this use case.
Tier Three: Agents with Full Internet Access represents what the team calls “YOLO mode” where any webpage can potentially perform prompt injection attacks against the agent. This tier requires domain allow lists, prompt injection mitigations, and the same isolation techniques as tier two.
The key insight is that a local agent in tier one and a browsing agent fleet in tier three might share the same tool like Chrome DevTools for Agents, but they should share nothing else about security model or access patterns. The team chose to maintain consent friction in tier one despite user requests because compromising trust for convenience creates unacceptable risk.
Chrome DevTools for Agents implements the Model Context Protocol, enabling it to work with any MCP-capable agent harness including Gemini CLI, Claude Code, Codex, and other platforms. The demonstration showed Gemini CLI opening Chrome, performing performance traces, analyzing results, making optimizations, and validating improvements - all autonomously through the MCP interface.
The project provides both MCP server and command line interface options, giving flexibility in how agents interact with the system while maintaining consistent functionality across integration patterns.
While the presentation emphasizes lessons learned, it’s clear this represents a production deployment being used by real developers and agents. Approximately 10% of the conference audience had already tried the tool, suggesting meaningful adoption within the developer community.
The team continues iterating on tool descriptions, error messages, and interface design as models evolve, treating this as an ongoing optimization problem rather than a one-time design exercise. The visualization of tokens per successful outcome across different use cases suggests active monitoring and data-driven improvement cycles in production.
Throughout the presentation, Hablich repeatedly emphasizes that every design decision involves trade-offs rather than clear wins:
The engineering philosophy centers on measuring impacts, understanding trade-off spaces, and making data-informed decisions rather than assuming optimal solutions exist. The team maintains that effectiveness must be measured before optimizing for efficiency, since fuel efficiency is meaningless if you can’t reach your destination.
This case study illustrates several broader principles for production LLM systems:
Agent interfaces require fundamentally different design patterns than human interfaces, despite sharing functional goals. The schema is the UI for agents, making tool descriptions and error messages critical components rather than secondary concerns. Token economics and context windows are first-class constraints that shape architecture and interface design. Security models must account for the different trust boundaries that emerge when agents operate with varying degrees of autonomy and internet access.
The focus on measurable outcomes, particularly tokens per successful outcome, reflects mature LLMOps practice where operational metrics drive iterative improvement. The team’s willingness to maintain friction in the user experience when it serves trust and security goals demonstrates thoughtful production deployment that resists pressure to optimize purely for convenience.
Chrome DevTools for Agents represents a sophisticated example of building production agent infrastructure that balances effectiveness, efficiency, discoverability, security, and maintainability across evolving model capabilities and use cases.
Slack's Developer Experience team embarked on a multi-year journey to integrate generative AI into their internal development workflows, moving from experimental prototypes to production-grade AI assistants and agentic systems. Starting with Amazon SageMaker for initial experimentation, they transitioned to Amazon Bedrock for simplified infrastructure management, achieving a 98% cost reduction. The team rolled out AI coding assistants using Anthropic's Claude Code and Cursor integrated with Bedrock, resulting in 99% developer adoption and a 25% increase in pull request throughput. They then evolved their internal knowledge bot (Buddybot) into a sophisticated multi-agent system handling over 5,000 escalation requests monthly, using AWS Strands as an orchestration framework with Claude Code sub-agents, Temporal for workflow durability, and MCP servers for standardized tool access. The implementation demonstrates a pragmatic approach to LLMOps, prioritizing incremental deployment, security compliance (FedRAMP), observability through OpenTelemetry, and maintaining model agnosticism while scaling to millions of tokens per minute.
Manus, a general AI agent platform, addresses the challenge of context explosion in long-running autonomous agents that can accumulate hundreds of tool calls during typical tasks. The company developed a comprehensive context engineering framework encompassing five key dimensions: context offloading (to file systems and sandbox environments), context reduction (through compaction and summarization), context retrieval (using file-based search tools), context isolation (via multi-agent architectures), and context caching (for KV cache optimization). This approach has been refined through five major refactors since launch in March, with the system supporting typical tasks requiring around 50 tool calls while maintaining model performance and managing token costs effectively through their layered action space architecture.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
