This case study examines Pi, a minimalist coding agent harness, through two production deployments at Databricks and Shopify. Databricks conducted internal benchmarking on their multi-million line codebase and found that Pi combined with Claude Opus 4.8 achieved the highest pass rates while maintaining significantly lower costs than more complex alternatives like Claude Code and Codex—up to 3x less context per turn and 2x cost differences on identical model runs. Shopify leveraged Pi's extensible architecture to build pi-autoresearch, an autonomous optimization loop that delivered measurable performance improvements including 300x faster unit tests and 20% faster React component mounting. The case demonstrates that minimal harness design with strong extensibility can outperform feature-rich alternatives in both cost efficiency and task completion rates.
This case study presents a counter-intuitive approach to LLMOps through the lens of Pi, a minimalist coding agent harness developed by Earendil Inc. The material is clearly promotional in nature, being published by the vendor itself, which necessitates critical evaluation of the claims made. However, the case study is strengthened by referencing external validation from two major technology companies: Databricks and Shopify. Both organizations independently evaluated or extended Pi in production environments, providing third-party verification of the core architectural principles being advocated.
The fundamental thesis challenges a prevailing trend in LLMOps where complexity is pursued as a path to better performance. Instead, Pi advocates for minimalism—shipping with only four tools and keeping system prompts and tool definitions below 1,000 tokens. This represents a significant architectural decision in the LLMOps space, where the harness layer (the infrastructure that wraps and orchestrates LLM calls) is kept intentionally sparse, relying on extensibility rather than built-in feature richness.
Databricks conducted internal research titled “Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase” which provides the most quantitative evidence in this case study. This research is particularly valuable from an LLMOps perspective because it addresses a critical challenge: how to evaluate coding agents on real-world production codebases rather than public benchmarks that may be contaminated by training data or gaming.
Databricks created their own benchmark based on tasks their engineering team regularly performs on their actual multi-million line production codebase. This approach to evaluation represents a mature LLMOps practice—recognizing that external benchmarks may not reflect actual production performance and investing in custom evaluation frameworks aligned with real business needs. The use of internal, production-derived tasks helps avoid the saturation bias that affects public benchmarks where models may have been specifically optimized for those exact tests.
One of the most significant LLMOps insights from the Databricks study is the explicit separation of model performance from harness performance. This is a crucial distinction often overlooked in production LLM deployments. The study found that “the harness a model is called from dramatically impacts cost and quality” and that “simple harnesses like Pi performed best on our workloads.” This suggests that in production environments, the orchestration layer can be as important as the underlying model choice—a key principle for LLMOps practitioners.
The study reported that running the same model with the same “thinking effort” through different harnesses resulted in cost differences of more than 2x while quality remained constant. This finding has significant implications for production economics and challenges assumptions that more sophisticated harnesses necessarily deliver better value.
The case study introduces the concept of “context discipline” as a key performance differentiator. According to Databricks’ findings, Pi sent approximately 3x less context per turn compared to alternative harnesses, managed context more tightly by maintaining a smaller working set, and completed tasks in fewer runs. This represents a fundamental LLMOps optimization—token efficiency at the harness level, not just at the prompt level.
In production LLM systems, context management directly impacts both latency and cost. Every token sent to the model incurs cost, and larger contexts can increase processing time. The claim that Pi achieves better context discipline through minimalism suggests that aggressive defaults and extensive built-in instructions in other harnesses may actually be counterproductive, adding noise rather than value to the instruction hierarchy.
When combined with Claude Opus 4.8 on the “xhigh” configuration (presumably a high-effort thinking mode), Pi achieved the highest overall pass rate while maintaining significantly lower costs than both Claude Code and Codex. While the source material doesn’t provide specific numerical pass rates or cost figures, the claim is that Pi outperformed on both dimensions simultaneously—a rare outcome that typically involves tradeoffs.
The study also noted that end-to-end engineering economics matter more than price-per-token metrics. An interesting observation was that running complex workflows on Claude Haiku 4.5 (a smaller, cheaper model) was often more expensive than Sonnet 4.6 (larger, more expensive) when code execution was involved, simply because the smaller model required more turns to complete tasks successfully. This reinforces the importance of total cost of ownership (TCO) analysis in LLMOps, where completion rates, number of turns, and task success must all factor into cost calculations, not just the sticker price of API calls.
The Shopify case study demonstrates a different aspect of LLMOps: the extensibility and customization requirements for production AI systems. Shopify Engineering, specifically David Cortés, built “pi-autoresearch” as a Pi extension to create an autonomous optimization loop for coding agents.
An intriguing aspect of the Shopify implementation is the meta-programming approach: they reportedly asked Pi to “create an extension for Autoresearch” and Pi read its own extension documentation to build the new workflow. This represents an advanced LLMOps pattern where the AI system is used to extend itself by referencing its own documentation. While this is compelling from a developer experience perspective, it also raises questions about the stability and predictability of such self-referential systems in production—a concern the source material doesn’t address.
The pi-autoresearch extension implements an autonomous experimentation loop that runs tests to identify what changes improve performance and what causes regressions. This represents a form of automated optimization that could be valuable in production environments where measurable performance targets exist. The system can theoretically self-improve by discarding regression-causing changes and keeping improvements.
Shopify reported several specific performance improvements:
These claims should be evaluated with appropriate skepticism given the promotional nature of the source. The 300x improvement in unit test speed, while impressive if true, lacks context about what baseline was being compared or whether this represents optimization of tests themselves or the systems being tested. The 20% React mounting improvement is more modest and potentially more credible. Without access to the original Shopify Engineering post or methodology details, it’s difficult to validate these specific numbers.
The broader point Shopify’s case illustrates is the value of extensibility in production LLM systems. Rather than shipping with every possible tool and feature, Pi’s architecture assumes users know their workflows best and provides the infrastructure to build custom extensions. This is a valid architectural choice that trades immediate out-of-box functionality for long-term flexibility and reduced bloat.
From an LLMOps perspective, this approach has several implications:
Pi’s core design philosophy centers on minimalism: shipping with only four tools and keeping the system prompt and tool definitions under 1,000 tokens. This stands in contrast to comprehensive harnesses that attempt to provide extensive built-in functionality. The argument is that most work can be accomplished with basics, and additional complexity should be added only when needed.
From a production LLMOps standpoint, this approach has several potential advantages:
However, there are also potential drawbacks that the source material doesn’t explore:
The concept of “context discipline” emerges as a key theme. This refers to the practice of managing context conservatively—not changing it without explicit user request, maintaining a stable prompt prefix, and avoiding unnecessary context expansion. For production LLM systems, this discipline has several benefits:
Cost management: Every token costs money at scale. A system that sends 3x less context per turn will have significantly lower operational costs in high-volume production environments.
Latency optimization: Smaller contexts process faster, reducing end-user latency. This is particularly important for interactive coding agents where developers expect rapid feedback.
Prefill efficiency: The source notes that local models with longer prefill times especially benefit from stable prompt prefixes. By maintaining context discipline, the system avoids expensive re-prefilling operations that can take minutes on local deployments.
Working set management: Keeping a tighter working set of relevant information helps the model focus on what matters rather than being distracted by extensive context.
The source material addresses an interesting historical argument: that “native” harnesses (presumably those built by the model providers themselves, like Claude Code from Anthropic) had a structural advantage because models were built around them. However, it argues this advantage has weakened as frontier models have become more generally competent at understanding terminal-style coding environments.
The material cites Anthropic recently cutting down Claude Code’s system prompt by 80% as evidence of this shift. If true, this represents a significant evolution in LLMOps thinking—moving from highly specialized, vendor-specific orchestration to more universal, environment-based interfaces.
The argument is that the question has shifted from “how native is the harness?” to “how well does it handle context to avoid redundancy and provide clean primitives?” This reframing suggests that clean architectural principles may now matter more than tight model-harness integration—a notable claim for LLMOps practitioners evaluating different orchestration approaches.
The case study makes specific claims about Pi’s suitability for local model deployment, which is an important LLMOps consideration as organizations increasingly explore on-premise and edge deployment for cost, privacy, or latency reasons.
Local models typically have:
Pi’s minimal default system prompt, small tool set, and context discipline are positioned as advantages in this environment. The stable prompt prefix is particularly valuable because it enables prompt caching and avoids the costly re-prefilling that can take minutes on local deployments.
This represents thoughtful consideration of deployment environment diversity in LLMOps—recognizing that production systems may need to run on various infrastructure, not just cloud APIs.
As promotional material from the vendor, this case study should be evaluated critically. Several important questions remain unanswered:
Validation and reproducibility: While Databricks and Shopify are cited, we don’t have access to the full studies, methodologies, or reproducible benchmarks. The Databricks study is referenced but not linked or published externally in the source material.
Selection bias: Did Databricks test Pi because they were already predisposed to minimalist approaches? Were there other harnesses tested that aren’t mentioned?
Complexity threshold: At what point does the extensibility model break down? For very complex workflows requiring dozens of custom tools, is the development overhead worth the minimal base?
Comparison fairness: When comparing Pi to “Claude Code and Codex,” are we comparing equivalent configurations? Were the other harnesses given the same opportunity for optimization?
Performance variance: The material reports aggregate results but doesn’t discuss variance. Were there task categories where Pi performed poorly?
Production maturity: How long have these systems been in production? What operational challenges have emerged?
Governance and safety: With extensibility comes the risk of unsafe or poorly-performing custom tools. How is this managed in production?
Despite the promotional framing, this case study surfaces several valuable LLMOps principles:
Harness architecture matters: The orchestration layer is not just plumbing—it has measurable impact on cost and quality. Production teams should evaluate harnesses as carefully as models.
Context is expensive: Token efficiency isn’t just a nice-to-have; it’s a core cost driver at scale. Systems that manage context discipline can deliver significant operational savings.
Extensibility vs. completeness tradeoffs: There’s a genuine architectural choice between shipping comprehensive functionality and providing extensibility frameworks. Neither is universally better; the choice depends on organizational context.
Custom benchmarking is essential: Databricks’ creation of internal benchmarks based on real tasks reflects mature LLMOps practice. Public benchmarks may not reflect actual production performance.
Total cost of ownership thinking: Evaluating LLM systems requires looking beyond per-token costs to completion rates, number of turns, development overhead, and maintenance burden.
Model-agnostic design: As models improve and commoditize, the value may shift to the orchestration layer. Harnesses that work across different models may have strategic advantages.
The case study ultimately presents a coherent architectural vision for production LLM systems that prioritizes minimalism, extensibility, and context discipline over comprehensive built-in functionality. While the claims require independent validation and the approach won’t suit every organization, it represents a thoughtful alternative to the complexity-maximizing trends in the AI tooling space.
Poolside AI, a foundation model company focused on code generation, developed a comprehensive "Model Factory" system that enables them to train and deploy models from scratch to production in 5-8 weeks with a team of fewer than 70 researchers. Their approach treats model building as 90% engineering, emphasizing automation, reproducibility, and rapid experimentation (10,000-20,000 experiments per month). The result is the Laguna S model (118B parameters, 8B active), which demonstrates that smaller models with better behaviors—persistence, verification, and backtracking—can compete with models 10x their size, suggesting a path toward commoditized, open-weight foundation models.
This case study explores the evolution of software development using AI coding agents over an 18-month period, from late 2024 through 2025 and into 2026. The speaker, a developer at Oschlo, transitioned from traditional software engineering to building production systems primarily using coding agents like Claude Code, Aider, Codex, and Pi. The solution involved developing systematic workflows incorporating skills, deterministic tools, multi-agent orchestration, automated verification, and autonomous systems like a "sentinel" that monitors CI/CD pipelines and automatically creates pull requests. Results demonstrate that complex features can be built in hours instead of weeks, with one example showing an end-to-end feature built using 2 million tokens over 1 hour 45 minutes with minimal human intervention, though at significant token costs that are becoming a limiting factor for enterprise adoption.
Cognition, the company behind Devon, discusses their journey building production-ready autonomous coding agents that operate in cloud environments. The conversation with Walden Yan (Co-founder, CPO at Cognition) and Cole Murray (creator of Open Inspect) explores the architectural decisions, infrastructure challenges, and production considerations for deploying AI agents that can autonomously write, test, and merge code. They discuss the shift from local IDE-based AI assistants to background agents that work autonomously in cloud environments, the technical infrastructure required to support this paradigm (including VM management, sandbox security, and state management), and real-world use cases like automated incident response, customer support triage, and continuous security scanning. The discussion covers how Devon now contributes 80% of commits on Cognition's repositories (up from 16% in January), representing a fundamental shift in how engineering teams work with AI.
