## Overview This case study provides a detailed look into how GitHub evaluates AI models for their flagship AI coding assistant, GitHub Copilot. The article, authored by GitHub engineers Connor Adams and Klint Finley, focuses specifically on their offline evaluation processes—the tests conducted before any changes are made to their production environment. This is a significant LLMOps case study because it reveals the operational complexity behind running a multi-model AI product at scale, where the stakes are high given the millions of developers who rely on GitHub Copilot daily. GitHub Copilot has expanded to support multiple foundation models including Anthropic's Claude 3.5 Sonnet, Google's Gemini 1.5 Pro, and OpenAI's o1-preview and o1-mini models. The challenge of maintaining consistent quality, performance, and safety across this diverse set of models while enabling user choice represents a sophisticated LLMOps problem. ## The Evaluation Framework GitHub's approach to model evaluation combines automated testing at scale with manual evaluation for subjective quality assessment. This hybrid approach acknowledges that while automated tests enable evaluation across thousands of scenarios, some aspects of output quality require human judgment or at least more sophisticated evaluation mechanisms. The team runs more than 4,000 offline tests, with most integrated into their automated CI pipeline. This represents a substantial investment in evaluation infrastructure and demonstrates the importance GitHub places on systematic quality assurance for their AI features. Beyond automated testing, they also conduct internal live evaluations similar to canary testing, where they switch a portion of GitHub employees ("Hubbers") to use a new model before broader rollout. ## Containerized Repository Testing One of the most interesting aspects of GitHub's evaluation methodology is their containerized repository testing approach. They maintain a collection of approximately 100 containerized repositories that have all passed their CI test batteries. The evaluation process involves: - Deliberately modifying these repositories to fail their tests - Allowing the candidate model to attempt to fix the failing code - Measuring the model's ability to restore the code to a passing state This approach creates realistic, end-to-end evaluation scenarios that go beyond simple benchmark tasks. By using real codebases with actual CI pipelines, GitHub can assess how well models perform in conditions that closely mirror real developer workflows. The team generates as many different scenarios as possible across multiple programming languages and frameworks, and they continuously expand their test coverage including testing against multiple versions of supported languages. This breadth of testing is essential for a product that needs to support developers working across diverse technology stacks. ## Key Metrics for Offline Evaluation GitHub's evaluation metrics reveal what they prioritize when assessing model quality: For code completions, they measure the percentage of unit tests passed and the similarity to the original known-passing state. The unit test pass rate is a straightforward quality indicator—better models fix more broken code. The similarity metric is more nuanced, providing a baseline for code quality assessment even though they acknowledge that there may be better ways to write code than their original version. For Copilot Chat, they focus on the percentage of questions answered correctly from a collection of over 1,000 technical questions. This metric directly measures the accuracy of the conversational AI capabilities. Across both features, token usage serves as a key performance metric. Fewer tokens to achieve a result generally indicates higher efficiency, which has implications for both latency and cost in production environments. ## LLM-as-Judge Evaluation A particularly noteworthy aspect of GitHub's evaluation approach is their use of LLMs to evaluate other LLMs. While some of their 1,000+ technical questions are simple true-or-false queries that can be automatically evaluated, more complex questions require sophisticated assessment. Rather than relying solely on time-intensive manual testing, they use another LLM with known good performance to evaluate the answers provided by candidate models. This LLM-as-judge approach enables evaluation at scale while still capturing subjective quality aspects that simple automated tests might miss. However, GitHub acknowledges the challenges inherent in this approach: ensuring that the evaluating LLM aligns with human reviewers and performs consistently across many requests is an ongoing challenge. They address this through routine auditing of the evaluation LLM's outputs. ## Continuous Production Monitoring The evaluation framework isn't just for new model adoption—GitHub runs these tests against their production models daily. This continuous monitoring approach allows them to detect degradation in model quality over time. When they observe degradation, they conduct auditing to identify the cause and may need to make adjustments, such as modifying prompts, to restore expected quality levels. This daily testing of production models represents a mature approach to LLMOps, acknowledging that model behavior can drift or change over time and that ongoing quality assurance is essential rather than a one-time gate. ## Infrastructure Architecture GitHub's infrastructure design enables rapid iteration on model evaluation. They have built a proxy server into their infrastructure that the code completion feature uses. This architectural choice allows them to change which API the proxy calls for responses without any client-side changes. The ability to swap models without product code changes dramatically accelerates their ability to test new models and configurations. Their testing platform is built primarily with GitHub Actions, demonstrating how they leverage their own tools for AI operations. Results flow through systems like Apache Kafka and Microsoft Azure, and various dashboards enable exploration of evaluation data. This modern data pipeline approach ensures that evaluation results are actionable and accessible to decision-makers. ## Responsible AI Integration Safety and responsible AI considerations are deeply integrated into GitHub's evaluation process. Every model they implement—whether GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro—undergoes thorough vetting for both performance and whether it meets their standards. They test both prompts and responses for relevance (filtering non-code-related questions) and toxic content (hate speech, sexual content, violence, self-harm content). Their responsible AI evaluations include red team testing, and they apply many of the same evaluation techniques used for quality and performance to safety assessment. They also guard against prompt hacking and model baiting, recognizing that a production AI assistant will encounter adversarial inputs. ## Decision-Making Challenges The article honestly addresses the complexity of making adoption decisions based on evaluation data. While some decisions are straightforward—a model that performs poorly across all metrics is easy to reject—many scenarios involve tradeoffs. The example given of a model with higher acceptance rates but also higher latency illustrates these challenges. GitHub notes that there can be inverse relationships between metrics: higher latency might actually lead to higher acceptance rates because users see fewer suggestions overall. Understanding these relationships requires sophisticated analysis and judgment rather than simple threshold-based decisions. ## Limitations and Balanced Assessment While this case study provides valuable insight into GitHub's evaluation processes, it's worth noting that the article is relatively high-level and doesn't disclose specific performance numbers, benchmark results, or detailed methodology. It's also a first-party account, so claims about the rigor of their evaluation should be understood in that context. The article focuses primarily on offline evaluation and mentions online/canary testing only briefly. A complete picture of GitHub's LLMOps practices would include more detail on how they handle online experimentation, rollback procedures, and the transition from offline evaluation to production deployment. Nevertheless, the practices described—containerized repository testing, LLM-as-judge evaluation, daily production monitoring, and infrastructure enabling rapid model iteration—represent sophisticated LLMOps patterns that other organizations can learn from as they build their own AI evaluation pipelines.