Cognition's deployed engineering team describes how they operationalize Devin, their AI software engineering agent, across enterprise customers through a forward-deployed engineering model. The problem addressed is the gap between AI agent capabilities and real-world enterprise software engineering needs, which encompass not just code generation but testing, review, deployment, and maintenance across complex legacy codebases. Their solution involves embedding engineers directly with customers to map agent capabilities to specific business problems, creating automated workflows, and establishing feedback loops back to product development. Results include delivering 150% additional engineering capacity to embedded customers, reducing project delivery timelines by 82%, doubling PR output compared to single-point tools, and enabling enterprises like Nubank and major Latin American banks to complete large-scale migrations with 50-66% reduction in required effort.
Cognition operates Devin, an AI software engineering agent platform that evolved significantly from its initial 2024 release when it achieved 13% on SweepBench to become a comprehensive enterprise solution by 2026. The presentation by Gia, a deployed engineering lead, provides insight into how Cognition operationalizes AI agents in production environments through a distinctive forward-deployed engineering approach. The company offers multiple product surfaces including a CLI, IDE interface, and their flagship Devin Cloud agent, but the focus is on enterprise deployment methodology rather than specific product features.
The core insight is that successful AI agent deployment at enterprise scale requires more than just capable models—it demands a sophisticated organizational structure that bridges the gap between product capabilities and customer problems. Cognition’s forward-deployed engineering team serves as this bridge, embedding directly with customers to understand their specific contexts, orchestrate agent workflows, measure outcomes, and feed learnings back to product development.
A fundamental insight presented is that software engineering encompasses far more than just writing code. The talk identifies the full software development lifecycle as including building features, maintaining software, reviewing code, deploying systems, and ongoing maintenance. While code generation itself is characterized as a “mostly solved problem” where models with sufficient context engineering can produce needed code blocks, this typically represents only 20% of the actual work. The remaining 80% involves testing, reviewing, deploying, and maintaining code across enterprise environments.
This creates a significant challenge for AI agent deployment: without careful orchestration and direction, agents simply “token max”—burning computational resources and spending without producing tangible business outcomes. The complexity of enterprise software development means that generic agent deployment yields poor results. Legacy codebases, regulatory requirements, existing processes, and integration challenges all create friction that pure AI capabilities cannot overcome alone.
Cognition’s approach centers on forward-deployed engineers who possess what they call a “T-shaped” skill set—broad capabilities across people skills, business understanding, process knowledge, and customer engagement, with deep spikes in specific areas. The team hires from diverse backgrounds including product management, software engineering, and founder experiences, valuing either strong business sense or deep technical expertise that can be complemented with the other dimension.
The forward-deployed motion operates in two critical directions. First, these engineers deeply understand customer problem spaces by spending significant time on customer calls—described as four to five hours daily—to identify strategic initiatives with the highest business leverage. They then map Devin’s capabilities specifically to these problems rather than deploying agents generically. This involves setting up automation that responds to specific alerts and events without manual triggering, essentially creating orchestrated workflows tailored to each customer’s environment.
Second, and equally critical, these engineers map customer problems back to product capabilities. They serve as the highest-fidelity evaluation mechanism for the product, gathering real-world feedback daily from field deployments and translating this into actionable product improvements. This creates a feedback loop where each deployment informs and improves subsequent deployments.
The team uses a framework of maximizing the overlap between two circles: the domain of their product and the bucket of customer problems. Product-market fit represents this overlap, and forward-deployed engineering actively works to expand it from both directions—solving customer problems with existing capabilities while evolving the product to address common enterprise challenges.
A major theme is the shift from early approaches focused on maximizing token usage to current emphasis on measurable business outcomes. In 2024 and earlier, the key performance indicator was simply token usage, but by 2026, the focus has moved to delivery metrics that demonstrate true value. The presentation acknowledges that measuring return on investment for AI agents remains an unsolved problem that represents significant market opportunity.
Cognition tracks several key metrics for agent sessions. They measure total engineering hours generated versus productive engineering hours, allowing differentiation between activity and value. Specific proof points include a three-month embedded engagement that delivered approximately 150% additional headcount equivalent—described as having 150 extra coworkers working on a migration or development project. They claim an 82% reduction in delivery timelines when comparing metrics before and after Devin deployment, compressing sprint cycles and time-to-market significantly. Additionally, they report delivering almost double the number of pull requests compared to what engineers achieved with single-point tools or without agent assistance.
Internally at Cognition, over six months they shipped nearly an order of magnitude more high-quality, robust pull requests using their own agent despite being potentially behind on hiring. This serves as both validation of their approach and a practical example of the agent operating in production.
Several public case studies illustrate the application of Devin in enterprise production environments. Nubank, a major Latin American financial institution, had an ETL migration project staffed with 50 engineers. Cognition delivered the migration in approximately one-third of the planned timeline using Devin autonomously. Another large Latin American bank had a tax identification system migration that Cognition completed with half the required effort.
These examples highlight deployment in legacy codebases using outdated languages like COBOL and JCL—technologies that represent significant challenges because they are no longer widely learned or understood. The ability to operate across such complicated codebases demonstrates the importance of the orchestration and context engineering that forward-deployed engineers provide.
The Built card case study reports an order of magnitude improvement in PR acceptance rate, delivering 10x worth of engineering talent weekly and generating weekly output equivalent to over 10 engineers at the organization.
The presentation emphasizes several principles critical to successful LLMOps at scale. First is the concept of “intelligent orchestration with measurable outcomes” replacing the earlier era of indiscriminate token usage. This requires understanding when and how to deploy agents rather than simply maximizing their runtime.
Second is the distinction between making individual engineers 10x faster versus making entire organizations 10x faster, including non-technical stakeholders. This organizational-level impact differentiates comprehensive agent platforms from single-point tools like standalone CLIs or IDEs. The ability to affect the entire organization requires automation, integration with existing workflows, and accessibility beyond just technical users.
Third is the importance of embedding with customers to understand their specific contexts. Cognition demonstrates commitment to this through examples like deploying an engineer to Brazil for 10 months to live near a customer and ensure their success. This level of engagement enables the deep understanding necessary to configure and orchestrate agents effectively in complex enterprise environments.
Fourth is the feedback loop between field deployment and product development. Forward-deployed engineers are characterized as “heralds of change” and bridges between products and problems. They must identify which engineering challenges are common across enterprises versus unique to specific users, and determine when workarounds or bugs should become features. This de-risks the product roadmap by ensuring development priorities align with actual market needs.
Several implicit LLMOps challenges emerge from the presentation. The importance of context engineering is repeatedly emphasized—models need sufficient context to generate appropriate code, but providing this context in enterprise environments with large, complex codebases requires sophisticated tooling and understanding.
The evaluation challenge is explicitly acknowledged as unsolved. While Cognition tracks metrics like engineering hours, PR counts, and timeline reductions, the speaker notes that truly solving ROI measurement for AI agents represents a major market opportunity. This reflects the broader LLMOps challenge of establishing reliable evaluation frameworks for agentic systems where success is multidimensional and context-dependent.
The distinction between activity metrics and outcome metrics appears throughout the discussion. Early focus on token usage represented an activity metric that didn’t correlate reliably with business value. The shift to measuring delivery timelines, PR acceptance rates, and project completion demonstrates maturation toward outcome-based evaluation, though challenges remain in standardizing these measurements across diverse customer environments.
The automation and orchestration challenge is central to the forward-deployed engineering role. Setting up agents to respond to specific alerts and events without manual triggering requires deep understanding of customer workflows, integration capabilities, and failure modes. This orchestration layer represents significant engineering work that generic agent deployments cannot provide.
The presentation reveals organizational principles that enable Cognition’s LLMOps approach. Core values include being “down for the mission” and “shipping” with everyone functioning as go-to-market regardless of formal role. The emphasis on correctness and no ego in feedback loops enables rapid iteration between field deployments and product development.
The hiring philosophy for forward-deployed engineers emphasizes flexibility in backgrounds but requires either strong business acumen or deep technical expertise. The acknowledgment that customer engineering versus great customer engineering distinguishes those who solve problems from those who exhibit “relentless curiosity for why” and can communicate insights back to roadmap in ways that improve outcomes for all customers.
The metric of “relentlessly tied to the customer” reflects an understanding that enterprise AI agent success depends heavily on customer success and adoption. Making customers champions within their organizations by delivering demonstrable results creates a virtuous cycle of expansion and refinement.
While the presentation provides impressive metrics, several considerations warrant attention. The case studies presented are selective success stories, and we do not see failure modes, challenges encountered, or situations where the approach did not work well. The 82% timeline reduction and 150% capacity increase are dramatic claims that, while potentially accurate for specific contexts, may not generalize across all enterprise scenarios.
The evolution from the 2024 assessment that Devin was “only useful if desperate and out of ideas” to 2026 claims of being “actually good now” represents significant progress, but this rapid evolution also suggests the technology remains in flux. The forward-deployed engineering model itself, while sophisticated, represents a heavy investment of human expertise that may not scale economically across all customer segments.
The acknowledgment that ROI measurement remains unsolved is honest and important. Without standardized evaluation frameworks, comparing the effectiveness of different approaches or validating improvement claims across diverse contexts remains challenging. The reliance on metrics like PR counts and timeline reductions, while practical, may not capture important dimensions like code quality, maintainability, or technical debt.
The requirement for four to five hours daily of customer calls per deployed engineer suggests a high-touch model that works for large enterprise customers but may not extend to smaller organizations. The example of deploying an engineer to Brazil for 10 months illustrates commitment but also highlights the resource intensity of this approach.
Overall, the case study illustrates a sophisticated approach to enterprise AI agent deployment that addresses real LLMOps challenges through organizational structure, customer embedding, orchestration, and feedback loops. The results appear substantial within specific contexts, though the approach requires significant human expertise and investment to execute effectively.
1Password applied AI agents to refactor their multi-million-line Go monolith (B5) as part of evolving their Unified Access system to support both human and agent-driven workflows. They built an agentic toolchain that combined Go SSA analysis, SQL parsing, and DataDog integration to analyze dependencies, map domain ownership, and determine extraction order for service decomposition. The agents successfully automated a 3,000+ call site migration in hours and provided useful extraction sequencing, but struggled with complex service extraction tasks that required coordination across schema evolution, deployment sequencing, and shared data contracts. The team achieved 20-30% productivity improvements on complex tasks while learning that agents work best when producing deterministic artifacts from well-specified problems, with human oversight remaining critical for sequencing constraints and system boundaries.
OpenAI's Forward Deployed Engineering (FDE) team, led by Colin Jarvis, embeds with enterprise customers to solve high-value problems using LLMs and deliver production-grade AI applications. The team focuses on problems worth tens of millions to billions in value, working with companies across industries including finance (Morgan Stanley), manufacturing (semiconductors, automotive), telecommunications (T-Mobile, Klarna), and others. By deeply understanding customer domains, building evaluation frameworks, implementing guardrails, and iterating with users over months, the FDE team achieves 20-50% efficiency improvements and high adoption rates (98% at Morgan Stanley). The approach emphasizes solving hard, novel problems from zero-to-one, extracting learnings into reusable products and frameworks (like Swarm and Agent Kit), then scaling solutions across the market while maintaining strategic focus on product development over services revenue.
Cognition, the company behind Devon, discusses their journey building production-ready autonomous coding agents that operate in cloud environments. The conversation with Walden Yan (Co-founder, CPO at Cognition) and Cole Murray (creator of Open Inspect) explores the architectural decisions, infrastructure challenges, and production considerations for deploying AI agents that can autonomously write, test, and merge code. They discuss the shift from local IDE-based AI assistants to background agents that work autonomously in cloud environments, the technical infrastructure required to support this paradigm (including VM management, sandbox security, and state management), and real-world use cases like automated incident response, customer support triage, and continuous security scanning. The discussion covers how Devon now contributes 80% of commits on Cognition's repositories (up from 16% in January), representing a fundamental shift in how engineering teams work with AI.
