Atlassian developed Rovo Dev Code Reviewer, an AI-powered code review agent, to address bottlenecks in their manual code review process that were slowing down software development cycles. The system uses a three-stage approach combining structured prompting with Claude 3.5 Sonnet, an LLM-as-a-judge quality check for factual correctness, and a fine-tuned ModernBERT model to filter for actionable comments. Deployed across 1,900+ repositories over a year-long evaluation, the system demonstrated a 30.8% reduction in median PR cycle time, reduced human-written review comments by 35.6%, and achieved a 38.7% code resolution rate where AI-generated comments led to actual code changes, while maintaining a human-in-the-loop design philosophy.
Atlassian’s Rovo Dev Code Reviewer represents a significant production deployment of LLM technology aimed at addressing a common software engineering bottleneck: the time-consuming nature of manual code reviews. The case study documents how Atlassian developed and deployed an AI-powered code review system across their entire engineering organization, involving over 1,900 repositories. The solution went Generally Available in October 2025 after a successful beta period, and the research was accepted by ICSE 2026, a prestigious software engineering conference. This implementation is particularly notable for its human-in-the-loop design philosophy and its multi-stage quality control approach that addresses common LLM production challenges such as hallucination and generating non-actionable feedback.
The business context is clear: manual code review, while valuable for maintaining code quality and sharing knowledge, creates significant delays in shipping features. Engineers spend hours reviewing code, creating bottlenecks that slow deployment cycles. The challenge was to automate repetitive, context-independent parts of code review while preserving human judgment for complex architectural and design decisions. Atlassian needed a solution that would accelerate development without compromising their high engineering standards or data security requirements.
The Rovo Dev Code Reviewer system architecture demonstrates sophisticated LLMOps practices across multiple dimensions. At its core, the system employs a three-stage pipeline designed to generate high-quality, actionable code review comments while filtering out hallucinations and low-value feedback.
Stage 1: Context-Aware Comment Generation with Structured Prompting
The foundation of the system is a carefully engineered prompting strategy built on top of Anthropic’s Claude 3.5 Sonnet. The prompt engineering approach is notably sophisticated, incorporating several key components that demonstrate production-grade prompt design. The system assigns the LLM a specific persona, instructing it to act as an experienced Atlassian software engineer. This persona definition helps ground the model’s responses in the appropriate context and expertise level.
The prompts include explicit task definitions that clearly articulate what the LLM should accomplish for each specific pull request. Critically, Atlassian implements chain-of-thought reasoning, guiding the LLM through a step-by-step reasoning process before generating comments. This CoT approach helps ensure logical consistency and allows the model to “think aloud” through its analysis, which is particularly important for complex code review scenarios.
The structured review guidelines embedded in the prompts represent enterprise-level best practices for code, test files, and comments. These guidelines ensure alignment with Atlassian’s quality standards and help maintain consistency across reviews. The system also leverages rich contextual information, feeding the LLM with PR titles and descriptions, related Jira issue summaries and descriptions (providing business motivation and requirements), and the complete code changes in diff format. This comprehensive context enables the model to understand not just what changed, but why it changed and what business problem it addresses.
Notably, Atlassian emphasizes their use of a “zero-shot structured prompting approach” augmented with readily-available contextual information. This design choice was driven primarily by privacy and data security concerns—critical requirements for enterprise customers. By avoiding fine-tuning on customer code, they maintain stronger data isolation and security guarantees.
Stage 2: LLM-as-Judge for Factual Correctness
Recognizing that hallucination is a fundamental challenge when deploying LLMs in production, Atlassian implemented a second stage that acts as a quality gatekeeper. This stage uses an “LLM-as-a-Judge” component based on GPT-4o-mini (a more cost-effective model compared to Claude 3.5 Sonnet). This judge reviews every comment generated in stage one, specifically checking for factual correctness against the actual code changes.
This multi-model approach demonstrates sophisticated LLMOps thinking: using a cheaper model for validation tasks while reserving the more expensive, capable model for generation. The judge filters out hallucinated or factually incorrect comments before they reach engineers, ensuring that only valid and reliable feedback is posted. This stage is crucial for maintaining trust in the system—if engineers frequently encountered incorrect AI suggestions, they would quickly lose confidence in the tool.
Stage 3: Fine-Tuned ModernBERT for Actionability Filtering
The third stage addresses a more subtle quality issue: even factually correct comments can be unhelpful if they’re vague or non-actionable. Comments like “This needs improvement” or “Add a blank line here” may be technically correct but don’t provide meaningful value to developers. To filter out this noise, Atlassian developed a custom classification model based on ModernBERT, an encoder-only architecture known for memory efficiency and long context length capabilities.
This model was fine-tuned using a proprietary dataset of over 50,000 high-quality Rovo Dev-generated comments sourced from internal dogfooding. The training objective was to classify comments based on whether they led to code resolution—defined as triggering an actual code change. This resolution-based training signal is particularly clever from an LLMOps perspective: it uses actual developer behavior as ground truth rather than relying on subjective quality assessments. Comments that developers act upon are, by definition, actionable and valuable.
The choice of ModernBERT over other architectures appears driven by practical production considerations: memory efficiency is critical when processing many PRs at scale, and the ability to handle long contexts is essential for code review scenarios where understanding surrounding code is important.
Atlassian’s deployment philosophy emphasizes that Rovo Dev is explicitly not an autonomous agent but rather a “human-in-the-loop AI software development agent.” This design choice reflects a broader organizational philosophy about the future of teamwork where AI empowers rather than replaces human expertise.
In practice, Rovo Dev performs an initial “first pass” review and posts comments with suggestions and potential bugs, but the final decision to accept or decline any suggestion rests entirely with human reviewers. The system is integrated seamlessly into Bitbucket, Atlassian’s code repository platform, allowing engineers to accept AI suggestions with a single click. This low-friction integration is crucial for adoption—if accepting suggestions required complex workflows, developers would likely ignore them.
This human-in-the-loop approach provides several critical benefits from an LLMOps perspective. It builds trust by allowing engineers to verify AI suggestions before accepting them, maintains accountability by keeping humans as ultimate decision-makers, respects human autonomy in the development process, and creates a feedback loop where human decisions can be used to refine and improve the AI over time. The code resolution rate metric (whether a comment led to a code change) directly leverages this human feedback signal.
The evaluation of Rovo Dev Code Reviewer demonstrates production-grade LLMOps practices in measurement and validation. Atlassian conducted a large-scale online evaluation spanning over one year across more than 1,900 repositories. This represents a real-world deployment study rather than a controlled experiment, providing authentic evidence of impact in actual working conditions.
The key metrics tracked were carefully chosen to reflect both efficiency gains and quality maintenance. The primary finding was a 30.8% reduction in median PR cycle time, representing a substantial acceleration in development velocity. The system also reduced human-written review comments by 35.6%, demonstrating that it successfully offloaded routine review work from human engineers.
Critically, Atlassian measured the code resolution rate—the percentage of comments that led to actual code changes in subsequent commits. Rovo Dev achieved a 38.7% resolution rate, which the case study honestly acknowledges is lower than the 44.45% resolution rate for human comments. This transparent reporting of comparative performance demonstrates intellectual honesty and helps calibrate expectations about AI capabilities. The fact that nearly 40% of AI-generated comments led to code changes is still impressive and indicates substantial value delivery.
The analysis also revealed that certain comment types were more likely to be resolved: readability issues, bugs, and maintainability concerns were the most frequently acted upon. This finding provides valuable insight into where the AI adds the most value and could inform future optimization efforts.
Qualitative feedback from engineers was reportedly overwhelmingly positive, with specific praise for the system’s ability to find nuanced bugs and subtle errors like duplicate method names. This qualitative dimension complements the quantitative metrics and helps validate that the system is genuinely useful from the engineer’s perspective.
Throughout the case study, Atlassian emphasizes that privacy and data security were “paramount concerns” and “non-negotiable requirements” for enterprise customers. This drove several architectural decisions, most notably the choice to use zero-shot prompting rather than fine-tuning on customer code. By keeping the base model general and providing context through prompting, Atlassian avoids the data governance complexities and potential privacy risks associated with training on proprietary codebases.
This approach also provides deployment flexibility: the same base models can serve all customers without requiring customer-specific model versions, simplifying the MLOps infrastructure. However, it’s worth noting that the actionability filter (ModernBERT) was fine-tuned on internal Atlassian data, suggesting that some internal customization was acceptable where privacy constraints were less stringent.
The case study reveals a thoughtful approach to model selection that balances capability with cost. Claude 3.5 Sonnet is used for the primary comment generation task, where sophisticated reasoning and code understanding are critical. GPT-4o-mini is used for the factual correctness check, where the task is more straightforward and cost efficiency matters at scale. ModernBERT, a much smaller encoder-only model, handles the final actionability classification.
This multi-tiered model strategy represents mature LLMOps thinking: different tasks have different capability requirements and cost profiles, and using the most expensive model for everything would be wasteful. The architecture demonstrates how to combine multiple models with different strengths to build an efficient production system.
While the results are impressive, several considerations warrant attention when interpreting this case study. First, this is a case study from Atlassian about their own product, published on their blog to promote Rovo Dev. The incentives to present positive results are clear, and while the research was accepted by ICSE 2026 (lending academic credibility), readers should maintain appropriate skepticism.
The 30.8% reduction in PR cycle time is substantial, but the case study doesn’t provide detailed information about confounding factors. Was this measured against a control group of repositories not using Rovo Dev, or is it a before/after comparison? Before/after comparisons can be influenced by other organizational changes, seasonal patterns, or evolving team practices. The fact that this was a “large-scale online evaluation” suggests it may have involved the entire organization rather than a controlled experiment, which makes causal attribution more challenging.
The 38.7% code resolution rate for AI comments versus 44.45% for human comments reveals an important gap. While Atlassian frames the AI rate positively (and it is meaningful), it also shows that human reviewers are still substantially better at generating actionable feedback. The system is clearly valuable as an augmentation tool, but the results don’t suggest it can fully replace human review.
The case study doesn’t discuss false positive rates or cases where the AI generated technically correct but contextually inappropriate suggestions. In enterprise settings, even a small percentage of bad suggestions could erode trust if they waste developer time. The multi-stage filtering approach is designed to minimize this, but quantitative data on precision would strengthen the evaluation.
Cost information is notably absent. Running Claude 3.5 Sonnet and GPT-4o-mini across 1,900+ repositories for every PR likely involves substantial API costs. For organizations considering similar implementations, understanding the cost-benefit tradeoff would be valuable. Atlassian’s scale may make this economically viable in ways that wouldn’t apply to smaller organizations.
Atlassian indicates that their work is continuing, with plans to investigate advanced context enrichment techniques and improve the system’s agentic capabilities. This suggests potential evolution toward more sophisticated retrieval mechanisms (possibly RAG-based approaches to pull in relevant code context beyond the immediate diff) or multi-step reasoning capabilities where the agent could engage in dialog or perform multiple analysis passes.
Rovo Dev Code Reviewer represents a sophisticated production deployment of LLM technology that demonstrates several LLMOps best practices: careful prompt engineering with structured guidance and chain-of-thought reasoning, multi-stage quality filtering to address hallucination and actionability concerns, thoughtful model selection balancing capability and cost, human-in-the-loop design that maintains trust and accountability, large-scale evaluation with meaningful metrics tracked over extended periods, and privacy-conscious architecture using zero-shot approaches for customer code.
The quantitative results—30.8% faster PR cycles and 35.6% reduction in human review burden—represent meaningful productivity gains, even accounting for the promotional nature of the case study. The 38.7% code resolution rate demonstrates that the AI generates genuinely useful feedback, though the gap versus human performance (44.45%) shows continued room for improvement. The system exemplifies how LLMs can augment rather than replace human expertise in complex knowledge work, and provides a valuable reference point for organizations considering similar AI-assisted development tools.
Cosine, a company building enterprise coding agents, faced the challenge of deploying high-performance AI systems in highly constrained environments including on-premise and air-gapped deployments where large frontier models were not viable. They developed a multi-agent architecture using specialized orchestrator and worker models, leveraging model distillation, supervised fine-tuning, preference optimization, and reinforcement fine-tuning to create smaller models that could match or exceed the performance of much larger models. The result was a 31% performance increase on the SWE-bench Freelancer benchmark, 3X latency improvement, 60% reduction in GPU footprint, and 20% fewer errors in generated code, all while operating on as few as 4 H100 GPUs and maintaining full deployment flexibility across cloud, VPC, and on-premise environments.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
OpenDev is an open-source, command-line AI coding agent written in Rust that addresses the fundamental challenges of building production-ready autonomous software engineering systems. The agent tackles three critical problems: managing finite context windows over long sessions, preventing destructive operations while maintaining developer productivity, and extending capabilities without overwhelming token budgets. The solution employs a compound AI system architecture with per-workflow LLM binding, dual-agent separation of planning from execution, adaptive context compaction that progressively reduces older observations, lazy tool discovery via Model Context Protocol (MCP), and a defense-in-depth safety architecture. Results demonstrate approximately 54% reduction in peak context consumption, session lengths extending from 15-20 turns to 30-40 turns without emergency compaction, and a robust framework for terminal-first AI assistance that operates where developers manage source control, execute builds, and deploy environments.
