Amp developed a sophisticated infrastructure called "Orbs" to enable LLM agents to autonomously operate in remote development environments without human intervention. The problem addressed was enabling agents to perform complex development tasks (starting servers, logging in, taking screenshots, running tests) on headless remote machines where traditional local development workflows wouldn't work. Their solution involved creating ephemeral Debian-based environments pre-configured with development tools, implementing idempotent setup scripts, designing agent-specific authentication endpoints, establishing structured documentation through AGENTS.md files throughout the codebase, and optimizing logs and tooling for agent consumption. The results demonstrated that frontier LLMs could autonomously navigate complex development workflows, execute multi-step testing procedures, and debug issues without explicit instructions on how to accomplish tasks.
Amp has developed a comprehensive LLMOps infrastructure called “Orbs” that enables LLM agents to autonomously operate in remote, headless development environments. This case study, published in July 2026, describes how Amp has optimized their entire development stack—from infrastructure to documentation—to be “agent-friendly,” allowing frontier models to perform complex development tasks without explicit step-by-step instructions. The use case demonstrates a mature approach to deploying LLM agents in production development workflows, where agents can start servers, authenticate, run tests, take screenshots, and debug issues entirely on their own.
The fundamental challenge Amp addressed was the traditional assumption that development work requires local environments with direct human interaction. When scaling agent-based development to remote machines, typical obstacles include authentication flows (OAuth, 2FA, cookies), environment setup complexity, debugging without visual feedback, and the cognitive load of understanding where things are and how they work. Amp’s solution represents a holistic rethinking of development environments specifically optimized for agent consumption rather than human developers.
At the foundation of Amp’s LLMOps approach are “Orbs”—ephemeral, standardized development environments running Debian 12. Each Orb comes pre-configured with an extensive toolkit including authenticated GitHub CLI, PostgreSQL and Redis databases, tmux for session management, ffmpeg and ImageMagick for media processing, ripgrep for searching, Bun and Node.js runtimes, npm and pnpm package managers, and a tool called “agent-browser” for web interaction. The completeness of this base image is critical for LLMOps: agents don’t need to figure out how to install fundamental tools, reducing decision trees and potential failure modes.
The system prompt for agents operating in Orbs explicitly informs them about their environment and how to acquire additional tools, typically through simple apt-get install commands. This represents a key LLMOps principle: making the agent aware of its operational context reduces hallucination and improves task success rates. The ephemeral nature of Orbs (they sleep when unused and wake on demand) also demonstrates infrastructure efficiency considerations important in production LLM deployments where compute costs matter.
Amp implements a sophisticated bootstrapping system through .agents/setup and .agents/resume scripts. The .agents/setup file in their main repository is a 428-line Bash script that runs every time a fresh Orb is created, establishing the complete development environment. This script starts PostgreSQL with performance tuning specifically for ephemeral environments (fsync off, synchronous_commit off, autovacuum off), creates database users and seeds test data, installs mise (a tool version manager) to ensure exact Node and pnpm versions match the lockfile, runs frozen-lockfile package installation to guarantee reproducibility, and installs Python dependencies like Pillow that their tooling requires.
Critically, the setup script also generates orb-specific guidance written to ~/.config/amp/AGENTS.md, which agents read alongside repository documentation. This generated documentation informs agents about orb-specific considerations like using the tmux session named “dev” for long-lived background processes, how to access pre-seeded development users through magic login URLs, and where to save visual artifacts. The system snapshots the sandbox after setup and reuses that snapshot for up to 24 hours for new orbs, demonstrating performance optimization in LLMOps—reducing agent wait time improves overall throughput and user experience.
The .agents/resume script runs on every wake-up cycle, ensuring the environment is in the correct state after sleep periods. While currently minimal in their implementation (re-establishing networking connections), this hook point represents important production considerations for stateful services and long-running agent workflows.
A central innovation in Amp’s approach is their “dev-server” skill, which contains a script called ensure-dev-server.sh designed specifically for agent use. This script embodies the principle of idempotency crucial for reliable agent operations: it can reuse a healthy running server, restart a wedged one (listening but not responding), restart a healthy server when environment values have been updated, or start fresh if nothing is running. The agent never needs to diagnose the current state—just run the script and get a working server.
This design dramatically reduces the cognitive complexity for LLM agents. Rather than having agents reason through process management, port conflicts, and service health checks, the idempotent script handles all state transitions. The script writes port allocation information to .amp/dev-ports.json, which other scripts and agents read instead of relying on hardcoded localhost:2000 values. This dynamic port discovery enables multiple checkouts to run concurrent dev servers without collision, demonstrating how agent-friendly infrastructure benefits human developers too.
The dev server architecture shows mature thinking about LLMOps observability: all ports used by the server, inference services, and metrics endpoints are recorded in structured JSON that agents can programmatically consume. This eliminates a common failure mode where agents might attempt connections to wrong ports or outdated endpoints.
Authentication represents one of the most significant challenges in agent-based development workflows, and Amp’s solution through /__dev endpoints is particularly noteworthy. They created development-only HTTP endpoints specifically designed for agent use: /__dev lists available endpoints, /__dev/log-me-in/<email> signs the browser in as any local user without credentials, /__dev/log-me-out handles sign-out, /__dev/sudo establishes no-op passkey sessions to bypass passkey checks during testing, and /__dev/preflight returns a JSON readiness report detailing whether secrets are configured, server health, user workspace status, project existence, credit availability, API key validity, and CLI connectivity.
The /__dev/log-me-in endpoint is described as “truly a game changer” because it eliminates the OAuth maze, 2FA flows, redirects, and cookie management that would otherwise block agent progress. Instead of agents getting stuck trying to create new users or change passwords, they reliably obtain authenticated sessions through a simple URL pattern. The preflight endpoint is equally important for debugging: when something fails, agents can curl this endpoint and receive explicit JSON describing what’s missing rather than attempting to guess from error messages.
This authentication architecture demonstrates a key LLMOps principle: designing systems explicitly for programmatic agent access rather than forcing agents to simulate human interaction patterns. The restriction of these endpoints to development mode (locally and in orbs only) shows appropriate security boundaries.
Amp’s logging strategy is explicitly designed for agent consumption. The dev server logs to .amp/in/server.log, but crucially, these logs also contain browser console output forwarded and tagged with [browser] prefixes. This unified logging gives agents a single file to grep for understanding what’s occurring in browser sessions, eliminating the need to correlate multiple log sources.
The .amp/in directory serves as a “scratch pad inbox for agents”—a designated location where various services write logs and agents save artifacts like screenshots and recordings. This spatial organization of information reflects understanding that LLM agents benefit from consistent, predictable file system layouts. When the agent’s browser screenshot comes back blank, the case study notes that “the logs were one grep away and the skill docs had already anticipated my exact failure,” showing how logging strategy integrates with documentation to enable rapid agent debugging.
This observability approach represents mature LLMOps thinking: rather than requiring agents to navigate complex distributed tracing systems or learn observability tools, information is structured in ways that leverage agents’ native capabilities (reading files, grepping text).
Perhaps the most sophisticated aspect of Amp’s LLMOps approach is their documentation strategy through 41 AGENTS.md files distributed throughout their codebase. Agents read these files on-demand when accessing files in directories containing them, creating contextual documentation that scales with codebase complexity. The files cover architecture (what Amp is, how clients interact), deployment (where and how the server is deployed, which third-party providers are used), code conventions (what to do and avoid in different folders), historical knowledge (nasty bugs encountered and how to avoid them), and operational procedures (how to build, run, modify, and test everything).
The root AGENTS.md serves as the entry point, containing code style guidelines, security guidelines, infrastructure administration, version control usage (Git and jujutsu), testing best practices, important folder locations, references to other AGENTS.md files, and comprehensive development tooling documentation including build commands, dev server skills and scripts, log locations and formats, port usage, agent-browser skill usage, and storybook usage patterns.
This distributed documentation approach demonstrates important LLMOps considerations for context management. Rather than providing agents with a massive single document (which would consume token budgets and reduce relevance), the documentation is modular and contextually loaded. The case study mentions agents reading these files “on-demand,” suggesting intelligent context loading based on which code areas the agent is working in.
The documentation also explicitly targets agent consumption rather than human readers. For example, guidance tells agents to “measure screenshots with PIL instead of eyeballing pixels”—acknowledging agent capabilities and limitations while providing concrete alternatives. The note “(Mind you: that’s a note for your agent rather than yourself)” in the case study reinforces that these documents are agent-first artifacts.
Amp implements a “skills” abstraction for packaging agent capabilities. The dev-server skill mentioned contains a SKILL.md file and associated scripts, representing a pattern for bundling related functionality. The skill documentation describes what the scripts will do, when to use them, and what outputs to expect. This skills architecture represents a pattern emerging in LLMOps: rather than expecting agents to compose complex operations from scratch every time, pre-packaged skills provide reliable, tested implementations that agents can invoke.
The case study mentions agents use an “agent-browser skill and tool” for web interaction, suggesting a sophisticated browser automation capability specifically designed for agent use. Combined with the authentication endpoints, logging integration, and screenshot capabilities, this represents a full-stack solution for agents to interact with web applications programmatically.
The case study includes a particularly insightful section where an agent describes its experience working in this environment. The agent notes “how rarely I have to guess” and describes “paved paths” for common operations. The agent emphasizes that setup, dev server, and seeding are all idempotent, and because the orb is disposable, the agent can “act, check, and correct instead of over-planning around mistakes I can’t observe.”
This mental model reveals key LLMOps success factors: reducing decision uncertainty, providing idempotent operations that allow trial-and-error approaches, designing for observability of actions and outcomes, making errors “cheap” through environment disposability, and anticipating failure modes in documentation. The phrase “the environment doesn’t just tolerate an agent; it assumes one, and tells it where the light switches are” encapsulates the philosophy: infrastructure explicitly designed for agent operation rather than human operation with agent access bolted on.
The case study demonstrates agents autonomously performing complex multi-step workflows without explicit how-to instructions. Examples include finding UI components in the codebase and generating screenshots in different configurations, and executing complete run-throughs of core product functionality including starting dev servers, logging in, creating resources, and capturing proof of completion. The author notes that six months prior, such capabilities would have seemed impossible for agents operating on headless remote machines.
The results section emphasizes that agents figured out procedures “by itself” from the codebase, tooling, documentation, and orb infrastructure—demonstrating that properly structured environments enable frontier models to exhibit significant autonomous capability. However, the case study maintains appropriate caution, noting “sometimes” agents miss local development environments “and less and less so,” suggesting ongoing refinement rather than complete replacement of traditional development workflows.
This case study represents a mature, production-grade approach to LLMOps that goes beyond simple prompt engineering or API integration. Several important patterns emerge for practitioners:
Infrastructure as Agent Interface: Rather than treating infrastructure as a human-facing concern with agent access as an afterthought, Amp designed their entire stack (Orbs, setup scripts, dev servers, authentication, logging) explicitly for agent consumption. This represents a paradigm shift in thinking about development infrastructure.
Idempotency and State Management: The emphasis on idempotent scripts that handle state transitions automatically is crucial for reliable agent operations. LLMs are better at deciding “what” to do than managing complex state machines, so infrastructure that handles the “how” enables more reliable agent behavior.
Observability Through Simplicity: Rather than complex observability tooling, Amp provides agents with simple text files to grep and structured JSON to parse. This leverages agents’ native text processing strengths rather than requiring them to learn specialized tools.
Documentation as Agent Context: The distributed AGENTS.md strategy shows sophisticated thinking about context management in LLM applications. By making documentation modular, contextual, and explicitly agent-targeted, Amp enables agents to access relevant information without overwhelming token budgets.
Authentication as a Solved Problem: The /__dev endpoints demonstrate that authentication doesn’t need to be an agent blocker. By providing programmatic access specifically for development/testing contexts, complex human-centric auth flows can be bypassed where appropriate.
The “Don’t Make Them Guess” Principle: The case study’s summary emphasizes reducing agent uncertainty. This principle—providing clear, consistent interfaces and explicit guidance—appears repeatedly and seems central to their success.
However, the case study also reveals important caveats. This is explicitly a development/testing environment (the /__dev endpoints only work in development mode), so questions remain about how these patterns translate to production agent operations. The infrastructure investment is substantial (41 documentation files, 428-line setup scripts, custom browser automation tools, specialized endpoints), which may not be viable for all organizations or use cases. The case study comes from the company building the agent platform itself, potentially introducing bias toward positive framing and obscuring challenges or failures encountered along the way.
The reliance on frontier models is also significant—the opening states “the current generation of frontier models incredibly good at checking their own work,” suggesting these capabilities may not transfer to smaller or older models. The ephemerality of Orbs (disposable environments) enables the “cheap errors” philosophy but also means state doesn’t persist, which could be limiting for certain workflows.
Overall, this case study demonstrates sophisticated LLMOps practices that extend well beyond simple LLM API integration. The holistic approach—spanning infrastructure, tooling, documentation, authentication, and observability—represents one of the more comprehensive agent enablement strategies documented publicly. For organizations looking to deploy LLM agents in complex technical environments, the patterns here (idempotent operations, explicit documentation, simplified authentication, unified logging, agent-aware infrastructure) provide a valuable blueprint, even if the full implementation requires significant engineering investment.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
This case study examines Cursor's implementation of reinforcement learning (RL) for training coding models and agents in production environments. The team discusses the unique challenges of applying RL to code generation compared to other domains like mathematics, including handling larger action spaces, multi-step tool calling processes, and developing reward signals that capture real-world usage patterns. They explore various technical approaches including test-based rewards, process reward models, and infrastructure optimizations for handling long context windows and high-throughput inference during RL training, while working toward more human-centric evaluation metrics beyond traditional test coverage.
Ramp built Inspect, a custom background coding agent that writes and verifies code in isolated cloud-based environments. The system addresses the need for faster, more powerful development workflows by running sessions in sandboxed VMs on Modal with full development environments, integrated with production tools like Sentry, Datadog, and GitHub. Within months of deployment, approximately 30% of all pull requests merged to frontend and backend repositories were written by Inspect, demonstrating rapid internal adoption through voluntary usage rather than mandate. The platform enables unlimited concurrent sessions, supports multiple interaction modes (Slack, web, Chrome extension), includes multiplayer collaboration, and provides both automated code generation and verification capabilities.
