## Overall Summary Uber's uReview represents a sophisticated production deployment of LLMs for automated code review at massive scale. The platform analyzes over 90% of approximately 65,000 weekly code changes (referred to as "diffs" at Uber) across six monorepos covering Go, Java, Android, iOS, TypeScript, and Python. The system was built to address a fundamental operational challenge: as code volume increased—particularly with AI-assisted code development—human reviewers became overwhelmed, leading to missed bugs, security vulnerabilities, and inconsistent enforcement of coding standards. The result was production incidents, wasted resources, and slower release cycles. What makes uReview particularly noteworthy from an LLMOps perspective is its explicit focus on combating the primary failure mode of AI code review systems: false positives. The team identified two distinct sources of false positives—LLM hallucinations that generate factually incorrect comments, and technically valid but contextually inappropriate suggestions (such as flagging performance issues in non-performance-critical code). The architecture and operational approach were specifically designed to achieve high precision, with the understanding that developer trust is fragile and quickly eroded by low-quality suggestions. The system maintains a 75% usefulness rating from engineers and sees 65% of its comments addressed—both metrics that reportedly exceed human reviewer performance at Uber (internal audits showed only 51% of human comments considered bugs and addressed). ## Technical Architecture and Multi-Stage Pipeline The core of uReview is a modular, prompt-chaining-based architecture that decomposes the complex task of code review into four distinct sub-tasks, each of which can evolve independently. This design choice is central to the system's LLMOps maturity, enabling targeted iteration and improvement without requiring wholesale system redesigns. The pipeline begins with **ingestion and preprocessing**. When a developer submits a change on Uber's Phabricator code review platform, uReview first filters out low-signal targets including configuration files, generated code, and experimental directories. For eligible files, the system constructs structured prompts that include surrounding code context such as nearby functions, class definitions, and import statements. This context engineering is critical for enabling the language model to produce precise and relevant suggestions rather than generic advice. The system employs a **pluggable assistant framework** where each assistant specializes in a specific class of issues. Currently, three assistants are in production: the Standard Assistant detects bugs, incorrect exception handling, and logic flaws; the Best Practices Assistant enforces Uber-specific coding conventions by referencing a shared registry of style rules; and the AppSec Assistant targets application-level security vulnerabilities. This modularity allows each assistant to use customized prompts and context, and to be developed, evaluated, and tuned independently—a key architectural pattern for managing complexity in production LLM systems. After comment generation, the system runs through extensive **post-processing and quality filtering**, which the team identifies as the key mechanism for achieving high precision. This multi-layered filtering process includes: (1) a secondary prompt that evaluates each comment's quality and assigns a confidence score, with thresholds customized per assistant type, programming language, and comment category based on developer feedback and evaluation data; (2) a semantic similarity filter that merges overlapping suggestions to avoid redundancy; and (3) a category classifier that tags each comment (e.g., "correctness:null-check" or "readability:naming") and suppresses categories with historically low developer value. The team notes that this filtering infrastructure is "just as important as prompts" for system reliability. ## Model Selection and Evaluation Uber conducts systematic empirical evaluation of different LLM configurations using a curated benchmark suite of commits with annotated ground-truth issues. They compute standard precision, recall, and F1 scores to identify optimal model combinations. The most effective configuration pairs Anthropic Claude-4-Sonnet as the primary comment generator with OpenAI o4-mini-high as the review grader, achieving the highest F1 score across all tested setups and outperforming OpenAI GPT-4.1, O3, O1, Meta Llama-4, and DeepSeek R1. The runner-up configuration was Claude-4-Sonnet paired with GPT-4.1 as the grader, scoring 4.5 points below the leader. Notably, they use different models for different stages of the pipeline—generation versus grading—optimizing for the specific strengths of each model at each task. The team emphasizes that they "periodically evaluate newer models using this approach and use the model combination with the highest F1 score," indicating an ongoing process of model evaluation and selection as new models become available. This systematic approach to model selection, grounded in quantitative benchmarks rather than vendor claims or anecdotal evidence, exemplifies mature LLMOps practice. ## Feedback Collection and Continuous Improvement uReview implements comprehensive feedback collection mechanisms that are tightly integrated into the developer workflow. Developers can rate each comment as "Useful" or "Not Useful" and optionally add explanatory notes. All comments, along with associated metadata including assistant origin, category, confidence score, and developer feedback, are streamed to Apache Hive via Apache Kafka. This data infrastructure supports long-term tracking, experimentation, and operational dashboards. The system employs both **automated and manual evaluation methods**. For automated evaluation, the team developed a clever technique: they re-run uReview five times on the final commit to determine if a posted comment was addressed. Because LLM inference is stochastic, a single re-run might miss a lingering issue or incorrectly flag one that's already fixed. Five runs was determined to be "the minimal count that virtually eliminates missed detections while keeping cost and latency low." A comment is considered addressed if none of the re-runs reproduce a semantically similar comment (with adjustments for when referenced code is deleted). This automated approach runs on thousands of commits daily in production, providing high-volume feedback signals. For manual evaluation, they maintain a curated benchmark of commits with known issues, used to evaluate precision, recall, and F1 scores against human-labeled annotations. The advantage of this approach is that it allows local iteration and testing before deploying new features, complementing the production-scale automated feedback. The feedback data enables granular tuning at multiple levels: per assistant type, per programming language, and per comment category. Files that received negative comments become benchmark cases that future versions should avoid. The category classification system allows automatic elimination of historically low-value categories within Uber's specific context. ## Operational Performance and Impact From an operational standpoint, uReview demonstrates impressive scale and performance. It processes over 10,000 commits weekly (excluding configuration files) with a median latency of 4 minutes as part of Uber's CI process. The team estimates that having a second human reviewer look for similar issues would require 10 minutes per commit, translating to approximately 1,500 hours saved weekly, or nearly 39 developer years annually. Beyond pure time savings, the system provides feedback within minutes of a commit being posted, allowing authors to address issues before human reviewers are engaged. The system's precision focus has paid off in adoption and trust. The sustained usefulness rate above 75% across all deployed generators, combined with the 65% automated address rate, significantly exceeds their internal benchmarks for human reviewer performance. This superior performance on these metrics is particularly noteworthy given that AI systems often struggle to match human judgment in nuanced tasks. ## Build-vs-Buy Decision and Third-Party Tool Comparison Uber explicitly addresses why they built in-house rather than using third-party AI code review tools, providing valuable insight into the considerations for production LLM systems. Three main factors drove this decision: First, most third-party tools require GitHub, but Uber uses Phabricator as its primary code review platform. This architectural constraint limited deployment options for off-the-shelf solutions tightly coupled with GitHub. Second, evaluation of third-party tools on Uber code revealed three critical issues: many false positives, low-value true positives, and inability to interact with internal Uber systems. uReview avoids these through its precision focus, integrated feedback loops, specialization to Uber's specific needs, and ability to access internal systems. Third, at Uber's scale (65,000 diffs monthly), the AI-related costs of running uReview are reportedly an order of magnitude less than typical third-party tool pricing. This cost consideration is significant and highlights how at sufficient scale, the economics of in-house development can become compelling even for complex AI systems. ## Key LLMOps Lessons and Tradeoffs The team shares several hard-won lessons that provide valuable guidance for production LLM deployments: **Precision over volume is paramount.** Early development revealed that comment quality matters far more than quantity. Developers rapidly lose confidence in tools that generate low-quality or irrelevant suggestions. The focus on fewer but more useful comments—through confidence scoring, category pruning, and deduplication—preserved developer trust and drove adoption. This represents a fundamental tradeoff in LLM system design: maximizing recall might catch more issues, but at the cost of developer engagement and long-term utility. **Built-in feedback is essential, not optional.** Real-time developer feedback proved critical for tuning. Simple rating links embedded in every comment, combined with automated evaluation of which comments were addressed, enabled feedback collection at scale directly from users. Linking this feedback to metadata (language, category, assistant variant) uncovered granular patterns enabling targeted improvements. This feedback infrastructure represents significant engineering investment but is foundational to the system's ability to improve. **Guardrails are as important as prompts.** Even with high-performing models, single-shot prompting was insufficient. Unfiltered outputs led to hallucinations, duplicates, and inconsistent quality. The multi-stage chained prompt approach—one step for generation, another for grading, others for filtering and consolidation—proved essential. The team emphasizes that "prompt design helped, but system architecture and post-processing were even more critical," challenging the common narrative that prompt engineering alone is sufficient for production LLM systems. **Category-specific performance varies dramatically.** Developers consistently disliked certain comment categories from AI tools: readability nits, minor logging tweaks, low-impact performance optimizations, and stylistic issues all received poor ratings. In contrast, correctness bugs, missing error handling, and best-practice violations—especially when paired with examples or links to internal documentation—scored well. This finding has important implications for system design: rather than trying to improve performance on all categories equally, focusing on high-signal categories and suppressing low-signal ones proved more effective. **Current limitations around system design and context.** The team candidly notes that uReview currently only has access to code, not other artifacts like past PRs, feature flag configurations, database schemas, or technical documentation. This limits its ability to correctly assess overall system design and architecture. It excels at catching bugs evident from source code analysis alone. However, they anticipate this may change with MCP (Model Context Protocol) servers being built to access these resources, suggesting awareness of the system's current boundaries and a vision for future enhancement. **Strategic deployment approach.** The gradual rollout—one team or assistant at a time, with precision-recall dashboards, comment-address-rate logs, and user-reported false-positive counts at each stage—enabled rapid, data-driven iteration while limiting the scope of potential regressions. Early users provided precise feedback that could be correlated with metrics to make objective go/hold decisions for releases. When issues surfaced (noisy stylistic suggestions, missed security checks), they A/B-tested fixes, tuned thresholds, and shipped improvements within a day. This approach built credibility and allowed the system to scale with confidence, demonstrating sophisticated change management for production AI systems. ## Architectural Decisions and Tradeoffs Several architectural decisions reveal important tradeoffs in production LLM systems: **CI-time review versus IDE-integrated review.** The team explicitly chose to perform AI reviews at CI time on the code review platform rather than relying solely on IDE or editor integrations. Their reasoning: less control over what developers do locally—they may not use AI review features or may ignore warnings. This parallels the established practice of running builds and tests at CI time even when available locally. This decision prioritizes consistent enforcement over convenience, though it may mean some issues are caught later in the development cycle. **GenAI versus traditional linters for best practices.** The team takes a nuanced view on when to use LLMs versus static analysis tools. For simple or syntactic patterns, linters remain preferable—they're accurate, reliable, and cheap. However, some properties require semantic understanding that's hard to encode in linters. Their example: the Uber Go style guide recommends using the `time` library for time-related operations, which requires understanding that a particular integer variable represents time—something LLMs handle much better than pattern-based tools. This suggests a complementary approach rather than wholesale replacement of existing tooling. **Pluggable architecture for extensibility.** The decision to build a pluggable assistant framework with specialized components for different review categories enables independent development, evaluation, and tuning of each assistant. This modularity comes with complexity costs in orchestration and consistency, but provides flexibility for the system to evolve different capabilities at different rates based on their effectiveness and value. ## Future Directions and System Evolution The team identifies several areas for expansion: support for richer context, coverage of additional review categories like performance and test coverage, and development of reviewer-focused tools for code understanding and risk identification. The emphasis on "keeping engineers firmly in control" reflects a design philosophy of augmentation rather than replacement of human judgment. The high "product ceiling" they identify for AI code review, combined with the large scope of potential impact, suggests continued investment and evolution. The existing feedback infrastructure, model evaluation process, and modular architecture position the system well for incremental improvement and expansion. ## Critical Assessment and Balanced Perspective While the case study presents impressive results, several considerations warrant balanced assessment: The 75% usefulness rating, while strong, means 25% of comments are not useful—a significant volume at their scale. The comparison to human reviewers (51% address rate) is interesting but may not be entirely apples-to-apples, as human reviewers often provide broader feedback including design suggestions that may not manifest as code changes in the same changeset. The "order of magnitude" cost savings versus third-party tools is claimed but not substantiated with specific figures. Build-vs-buy decisions involve not just API costs but development, maintenance, and opportunity costs that aren't detailed. The system's current limitations around accessing broader context (documentation, schemas, past PRs) represent significant constraints on its ability to provide truly comprehensive review. The aspiration to address this through MCP servers is forward-looking but speculative. The time savings calculation (39 developer years annually) assumes that without uReview, these issues would require 10 minutes per commit of additional human review time. However, some of these issues might have gone undetected even with traditional review, or might have been caught later in testing, making the counterfactual somewhat uncertain. Nonetheless, the case study represents a mature, production-scale deployment of LLMs with sophisticated approaches to quality control, feedback integration, continuous evaluation, and developer experience. The team's willingness to share specific challenges, limitations, and lessons learned enhances the credibility of their account and provides valuable guidance for others building production LLM systems.