## Overview Codeium is an AI developer tools company that builds IDE plugins for code generation, autocomplete, chat, and search across 70+ programming languages and 40+ IDEs. The company claims over 1.5 million downloads and positions itself as the highest-rated developer tool in the Stack Overflow survey, even above tools like ChatGPT and GitHub Copilot. This case study, presented by Kevin (who leads their product engineering team), focuses on how Codeium approached the fundamental challenge of context retrieval for code generation at scale—and why they believe traditional embedding-based approaches are "stunting AI agents." The core thesis is that while embeddings have been the standard approach for retrieval-augmented generation (RAG) in code tools, they hit a performance ceiling that prevents truly accurate multi-document retrieval needed for high-quality code generation. Codeium's solution involves running parallel LLM inference across thousands of files rather than relying on vector similarity, enabled by their vertically integrated infrastructure that makes this computationally feasible. ## The Problem with Traditional Approaches The presentation outlines three main approaches to context retrieval for code generation: **Long Context Windows**: While expanding LLM context windows seems like an ergonomically easy solution, the latency and cost make it impractical. The example given is that Gemini takes 36 seconds to ingest 325,000 tokens—but a moderately sized repository easily exceeds 1 million tokens (~100K lines of code), and enterprises often have over a billion tokens of code. This makes long-context approaches infeasible for real-world enterprise use. **Fine-Tuning**: Training custom models per customer to reflect their code distribution requires continuous updates, is computationally expensive, and requires maintaining one model per customer. This is described as "prohibitively expensive for most applications." **Embeddings**: The standard RAG approach of converting code to vector embeddings for similarity search. While inexpensive to compute and store, embeddings struggle with reasoning over multiple items and have limited dimensional space to capture the full complexity of code relationships. ## Limitations of Embedding-Based Retrieval The presentation argues that embedding performance has plateaued. Looking at benchmarks over time, even the largest embedding models converge to approximately the same performance level (within ±5%). The fundamental issue, according to Codeium, is that you cannot distill all possible English queries and their relationships to code into a fixed-dimensional vector space. A concrete example illustrates the problem: when building a React contact form, effective code generation needs to retrieve: - Design system components (existing buttons, inputs) - Pattern matches with other forms in the codebase - Style guides (e.g., Tailwind classes) - Local and external documentation This multi-document retrieval challenge is poorly captured by standard benchmarks, which tend to focus on "needle in a haystack" scenarios—finding a single relevant document rather than assembling multiple relevant pieces. ## Product-Driven Evaluation A significant portion of the discussion focuses on how Codeium built evaluation systems that mirror real production usage rather than academic benchmarks. They use a metric called "Recall@50"—what fraction of ground-truth relevant documents appear in the top 50 retrieved items—which better reflects the multi-document nature of code retrieval. To build evaluation datasets, they mined pull requests and their constituent commits, extracting mappings between commit messages (English) and modified files. This creates an eval set that mimics actual product usage: given a natural language description of a change, can the system retrieve the relevant files? When they tested publicly available embedding models against this product-led benchmark, they found "reduced performance"—the models struggled with the real-world mapping between English commit descriptions and code relevance. ## The M Query Approach Codeium's solution, called "M Query," takes a fundamentally different approach: instead of computing vector similarity, they make parallel LLM calls to reason over each item in the codebase. Given N items and a retrieval query, they run an LLM on each item to determine relevance—essentially asking the model to make a yes/no judgment on each file. This approach provides "the highest quality, highest dimension space of reasoning" compared to low-dimensional vector embeddings. The system then compiles rankings considering: - Active files - Neighboring directories - Most recent commits - Current ticket/task context The key enabler for this computationally expensive approach is Codeium's vertical integration. ## Vertical Integration as an LLMOps Strategy Three pillars enable the M Query approach: **Custom Model Training**: Codeium trains their own models optimized for their specific workflows. This means when users interact with the product, they're using models purpose-built for code retrieval and generation, not general-purpose models. **Custom Infrastructure**: The company has roots in ML infrastructure (previously called "Exafunction"), and they've built their own serving infrastructure "down to the metal." This enables speed and efficiency that they claim is unmatched, allowing them to serve more completions at lower cost. **Product-Driven Development**: Rather than optimizing for research benchmarks, they focus on actual end-user results when shipping features. This means looking at real-world usage patterns and user satisfaction rather than local benchmark improvements. The claimed result is that their computation costs are 1/100th of competitors using APIs, enabling them to provide 100x the compute per user. This makes the parallel LLM reasoning approach economically viable even for free-tier users. ## Production Deployment and Results The M Query system was rolled out to a percentage of their user base (described as "small" but noting that with 1.5 million+ downloads, this still reached a significant number of users). Key production characteristics: **Performance Requirements**: The system must be fast—M Query runs thousands of LLMs in parallel to enable streaming code generation within seconds or milliseconds, not minutes or hours. This is critical for IDE integration where latency directly impacts developer experience. **Quality Metrics**: They track thumbs up/down on chat messages, code generation acceptance rates, and ultimately how much code is written for users. These product metrics, rather than offline benchmarks, drive iteration. **Observed Results**: Users in the test showed more thumbs up on chat messages, higher acceptance rates on generations, and more code being written for them overall. The presentation includes a demonstration of querying for "usage of an alert dialogue" and retrieving relevant source code from modified internal Shad CN components—showing the system successfully reasoning over files in monorepos and remote repositories. ## Iteration Cycle and Future Direction Codeium describes their iteration cycle as: - Starting with product-driven data and evaluation (derived from actual user behavior) - Applying compute-intensive approaches enabled by vertical integration - Pushing updates to users in real-time (overnight deployments) - Measuring production results (thumbs up/down, acceptance rates) - Iterating based on real-world feedback The "context engine" built around M Query is positioned as a foundation for future features beyond autocomplete and chat, including documentation generation, commit messages, code reviews, code scanning, and converting Figma designs to UI components. ## Critical Assessment While the presentation makes compelling arguments about embedding limitations, several caveats apply: - The claim of 1/100th cost compared to competitors is difficult to verify without access to their infrastructure and pricing - The "highest rated" status from Stack Overflow surveys may reflect factors beyond technical capability (e.g., the free tier offering) - The parallel LLM approach, while innovative, presumably has its own scaling challenges at very large codebases - Product-led evaluation, while valuable, may introduce its own biases toward existing user behavior patterns The autonomous driving analogy at the end—suggesting that more compute eventually solves hard problems—is presented as a thesis rather than a proven outcome for code generation. However, the approach of treating embeddings as a "heuristic" to be replaced by more compute-intensive methods as infrastructure costs decrease is a reasonable strategic bet. The vertical integration approach represents a significant investment moat but also a risk if foundation model improvements outpace custom development. The company's willingness to provide the same compute-intensive features to free users suggests confidence in their cost structure or a growth-focused strategy prioritizing market share.