Offline LLM Generation for Hyper-Personalized Carousel Recommendations

Overview

DoorDash’s case study presents a sophisticated LLMOps framework for generating personalized product carousels at scale across their grocery and retail verticals. The core innovation lies in their architectural decision to position LLMs as offline content generators rather than inline request-path services, thereby avoiding the latency, cost, and reliability challenges that plague most real-time LLM deployments. The system generates individualized merchandising surfaces—complete with custom titles, subtitles, and curated product selections—for millions of consumers, with each carousel tailored to a specific user’s purchase history and preferences.

The business problem DoorDash addresses is fundamental to modern e-commerce personalization: traditional recommendation systems can rank items effectively, but the actual presentation layer—the carousels, groupings, and thematic organization that guide discovery—typically comes from a fixed library of hand-curated templates. This creates a ceiling on personalization depth. While you might rank products differently for each user, everyone sees the same “Best Sellers” or “Popular Items” carousel headers. DoorDash’s framework breaks this constraint by making the merchandising surface itself a per-user generative artifact.

Architectural Foundation: The Consumer Memory Block

The technical foundation of DoorDash’s approach is what they call the “consumer memory block,” a structured, typed representation of everything known about an individual consumer. This primitive is deliberately engineered to be LLM-friendly, with several key properties that make it suitable for production use at scale. The memory block is composable, meaning different use cases can request different subsets of information—a dietary personalization use case needs different signals than a pet products use case. Each component, called a sub-block, maintains a stable schema and serializes to compact JSON that fits within LLM context windows.

Critically, the memory block is evidence-based rather than inferred. Every sub-block derives from observed consumer behavior and explicit signals with full provenance. This design choice enables the prompt to instruct the model to generate content only when actionable evidence exists and to abstain otherwise, which proves essential for maintaining quality at scale. The memory block includes long-running preferences, behavioral patterns, household context, brand affinities, and taxonomy-level purchase summaries. DoorDash positions this as one of their first systems to use consumer-level structured state as a typed input to an LLM, with the LLM’s output treated as a first-class artifact stored in retrieval infrastructure.

An important operational detail is the per-use-case trimming of memory blocks. A full consumer memory block contains far more information than any single use case needs, and DoorDash discovered through experimentation that including irrelevant context degrades LLM output quality while unnecessarily increasing token costs. They implement explicit allowlists of sub-blocks per use case, dropping everything else before the LLM sees the input. This preprocessing step happens upstream in their feature engineering platform, not during inference, which keeps the LLM pipeline simple and focused.

Multi-Stage Pipeline Architecture

The system architecture separates cleanly into offline write and online read paths, meeting at a vector index and metadata store. The write path handles all expensive LLM operations in batch mode, while the read path serves pre-computed artifacts through fast retrieval operations. This separation is the key to making generative personalization economically viable at DoorDash’s scale.

Offline Write Path

The write path begins with targeted cohort construction. Not every consumer benefits equally from LLM-generated carousels, and at DoorDash’s scale, unnecessary LLM invocations represent significant wasted cost. They push cohorting entirely upstream by creating a table of eligible consumers in their declarative feature engineering platform. This upstream job produces per-use-case trimmed memory block payloads as precomputed datasets that the LLM inference pipeline simply consumes. This design choice provides multiple benefits: different prompt experiments share the same cohort table without re-running expensive joins, the LLM pipeline doesn’t need to read from Apache Iceberg at inference time (removing a major class of failure modes), and memory block trimming happens once in a specialized, optimized location rather than repeatedly inside inference workers.

The core generative step is a single batch LLM call per consumer per use case. The prompt structure consists of a system prompt that defines the model’s role as a merchandising generator, specifies the JSON output schema, and encodes hard constraints around retrieval-friendly search intents, abstention behavior, and safety requirements. The user prompt injects the trimmed consumer memory block JSON and requests up to N carousels for the configured theme. The output schema is strict and machine-validated. Each carousel includes a title, subtitle, confidence score, and a list of search intents that will later become embedding-based retrieval queries.

A critical design element is that the model is explicitly instructed and allowed to abstain when insufficient consumer evidence exists. Abstention is a first-class output rather than a failure mode, which prevents generic, low-confidence outputs from polluting the index. DoorDash emphasizes that they treat prompt engineering as a measurement discipline rather than an art form. Every prompt change must move a metric on a held-out evaluation set. Their prompts went through more than ten production-evaluated revisions per use case, with each revision targeting specific failure classes surfaced by their evaluators—titles that were grammatical but not retrieval-friendly, search intents that drifted from the title’s semantic meaning, or carousels generated from weak evidence.

Scaling batch LLM inference to millions of consumers required substantial distributed systems engineering. The batch LLM API has a 24-hour completion window, which becomes the binding constraint when the eligible cohort reaches millions of users. Their production implementation uses Metaflow’s foreach construct to fan out into independent shards, each running as a self-contained Kubernetes pod with its own batch API budget. Three specific design choices enable this to work: they pass sharded payloads through object storage rather than serializing DataFrames as Metaflow artifacts (keeping the metadata service out of the data plane), they implement per-shard fault isolation so a failed shard can be independently retried without re-running successful shards, and they use vectorized iteration within workers rather than row-wise processing, achieving roughly order-of-magnitude speedup in per-shard parsing.

Before generated carousels reach online storage, they pass through deterministic quality gates including confidence score thresholds, minimum search intent counts, title deduplication per consumer, and structural cleanup of parallel arrays containing intents, taxonomy IDs, and filter tags. These filters are intentionally cheap and explainable, while more expensive LLM-based evaluators run on top of what passes these gates.

The embedding stage is a separate GPU-accelerated parallel flow that converts every generated search intent into a 256-dimensional vector using DoorDash’s internal embedding model. This step expands carousels into per-intent rows, batches embeddings, validates them (filtering NaN/Inf/zero vectors), and writes Parquet files sized for optimal bulk import into Milvus. Separating embedding from LLM inference allows DoorDash to iterate on the embedding model and LLM prompt independently.

The vector database architecture uses Milvus with several non-obvious design choices. The consumer_id serves as the partition key on the search-intent collection, meaning every serving-time query is scoped to a single consumer and partition-key routing ensures only the relevant segment is scanned rather than the whole collection. This caps per-request embedding-based retrieval cost as the cohort grows. They implement blue/green collections per use case, with each refresh writing to a fresh time-stamped collection. After bulk import completes and is verified, an alias swap atomically points production traffic at the new collection while the old collection is released. This provides safe, zero-downtime rollouts and trivial rollbacks by simply re-pointing the alias.

Online Read Path

At request time, the serving path involves no LLM calls—only retrieval and assembly operations. The feed service executes a DAG that transforms pre-computed carousel metadata into fully assembled carousels. It begins with a metadata lookup fetching carousel definitions for the consumer from the online metadata store, grouping by theme and ranking within each theme by the LLM’s confidence score. Per-theme experiment gating ensures consumers are only exposed to themes where they’re in the targeted store type, have carousel metadata, and are in the treatment arm of the A/B test.

The retrieval system is notably hybrid, running two parallel paths. For embedding-based retrieval, the service issues a consumer-partitioned Milvus query fetching all of the consumer’s search-intent embeddings, regroups them by carousel, and runs approximate nearest neighbor search against the in-store, in-stock item embedding collection scoped to the current submarket and business. A similarity threshold filters low-confidence matches. In parallel, structured taxonomy retrieval pulls items by IDs assigned during generation, composing structured filters like dietary qualifiers when applicable. When a carousel lacks the structured filters to make taxonomy retrieval safe, that branch is intentionally skipped and embedding-based retrieval alone owns the carousel.

The final fusion step merges embedding-based retrieval and taxonomy results per carousel, deduplicates by item, and packages everything into the final carousel object. Each item carries a source tag for downstream analysis showing which retrieval mode contributed, providing a feedback signal about where each path pulls its weight. DoorDash emphasizes that hybrid retrieval beats either path alone, with the source attribution giving them visibility into coverage contributions.

Evaluation Framework: LLM-as-Judge for Generative Recommendations

DoorDash identifies evaluation as the hardest part of shipping their generative recommendation system. Traditional recommendation metrics like click-through rate or conversion are too slow and noisy for the inner loop of prompt iteration, while human review doesn’t scale to per-consumer outputs. They built a hybrid offline evaluation pipeline combining deterministic rule-based checks with LLM-as-judge evaluators, forming a continuous integration suite through which every prompt revision must pass before online experimentation.

Every prompt revision is scored on a fixed-size, stratified sample filtered to production-quality outputs and stratified across confidence levels so revisions are comparable regardless of underlying confidence distribution shifts. Each sample carries a manifest including seed, filters, and distribution for full reproducibility. Rule-based evaluators are cheap, deterministic checks for structurally definable properties—does the title open with a recognized qualifier, does it close with a valid category at appropriate granularity, does the structural shape match downstream retrieval expectations. These run in seconds and catch the long tail of regressions that don’t require model evaluation.

LLM-as-judge evaluators handle properties requiring semantic reasoning. These are separate, smaller LLMs each with carefully designed rubrics, scoring aspects like whether the carousel’s qualifier actually matches the consumer’s evidenced preferences in the memory block, whether the title is a coherent and plausible concept (catching contradictions grammar checks miss), whether each generated search intent is consistent with the title’s qualifier and granularity, and whether assigned taxonomy IDs align with both the title and underlying memory block.

Each metric has a pre-defined launch threshold, and every threshold must be met before a prompt can be considered ready for online testing. This prevents the common failure mode where a prompt change improves one quality dimension while quietly regressing another. DoorDash considers this evaluation framework one of their most transferable contributions, arguing that any team using LLMs to generate user-facing artifacts needs similar offline evaluation infrastructure to iterate at engineering speed rather than experiment speed.

Production Impact and Results

The framework is in production generating per-consumer carousels across DoorDash’s New Verticals surfaces. Their example comparison illustrates the impact: a consumer in a household with two adult cats previously saw a generic “Best Sellers” carousel showing a mix of dog, cat, and small-animal products identical for every consumer. With the framework enabled, the same consumer sees a personalized “Cat Dry Food” carousel with title and retrieval intents generated offline based on their memory block, showing dry cat food SKUs actually in stock at their specific store. A three-week A/B experiment on retail pages showed approximately 1% increase in order rate for pet products and 0.6% increase in active users in the Pets category.

Critical LLMOps Lessons and Challenges

DoorDash surfaces several important lessons with broad applicability. They emphasize that architectural decoupling of generation from serving—treating the LLM as an offline content generator and the vector index as the serving layer—is what makes the system both fast and reliable at their scale. Inline LLM calls would have made the same product impossible at the queries-per-second and service-level objectives of high-traffic store pages.

They identify consumer state as the bottleneck rather than the model itself, with the richness and structure of input being the single biggest determinant of output quality. The typed, evidenced, composable consumer memory block is what unlocks meaningful per-consumer prompts. They stress that prompt engineering must be treated as a measurement discipline—without an offline evaluation suite, prompt changes are guesses; with one, they become versioned artifacts with measurable improvements. Building the evaluation framework first, even before achieving good prompts, was their highest-leverage decision.

The trimming of input before output proved critical, with per-use-case sub-block trimming providing meaningful token cost reduction and, more importantly, improved output quality by removing context that would otherwise consume model attention. They found that hybrid retrieval beats either path alone, with source attribution providing feedback about where each path contributes. When eligible cohorts exceed what fits in a single batch API window, they note that prompt engineering stops and distributed systems engineering begins—sharding, fault isolation, and out-of-band data passing all become requirements.

Future Directions

DoorDash outlines several active investment areas. They’re exploring merchant-conditioned generation, conditioning the LLM on merchant-side memory blocks in addition to consumer blocks so generated carousels reflect both what consumers want and what specific stores can credibly serve. The pipeline is theme-agnostic, making new themes an exercise in defining prompts, evaluation suites, and memory block trims without changing serving infrastructure. They’re investigating faster refresh cadences, with the current refresh representing a cost/freshness tradeoff and incremental refresh paths potentially allowing newly observed behavior to influence the next session. Multilingual generation through DoorDash’s internationalization stack is in progress, and they’re working on retrieval-augmented-generation style memory block selection to replace static per-theme allowlists with dynamic retrieval of the most relevant sub-blocks for each generation request.

Critical Assessment

While DoorDash presents compelling results, several aspects warrant careful consideration when evaluating this approach. The measured lift of approximately 1% in order rate, while positive and likely meaningful at DoorDash’s scale, represents incremental rather than transformational improvement. This suggests the framework is valuable but not revolutionary, which is actually a strength—it demonstrates production viability for realistic gains rather than making outsized claims.

The framework’s complexity is substantial, requiring coordination across batch LLM inference, distributed systems engineering, vector database management, hybrid retrieval, and sophisticated evaluation infrastructure. Organizations considering similar approaches should carefully assess whether they have the engineering capacity and scale to justify this investment. The write path alone involves Metaflow orchestration, Kubernetes pod management, sharded fault-tolerant processing, GPU-accelerated embedding, Milvus collection management with blue/green deployments, and coordination with feature engineering platforms. Smaller organizations might achieve 80% of the benefit with 20% of the complexity through simpler approaches.

The reliance on structured consumer memory blocks is both a strength and a potential limitation. While this structured approach enables reliable LLM conditioning and quality guarantees, it requires significant upstream investment in consumer modeling infrastructure. The memory block itself represents a substantial engineering artifact that must be maintained, versioned, and kept synchronized with evolving understanding of consumer behavior. Organizations without mature consumer data platforms may find this prerequisite challenging.

The evaluation framework, while sophisticated and appropriate for production systems, adds substantial operational overhead. Maintaining both rule-based evaluators and LLM-as-judge systems, defining and monitoring launch thresholds, and managing stratified evaluation samples requires dedicated investment. DoorDash’s emphasis on treating this as continuous integration for prompts is correct, but organizations should recognize this transforms prompt engineering from experimentation to software engineering with corresponding process and tooling requirements.

The hybrid retrieval approach combining embedding-based and taxonomy-based retrieval is pragmatic and addresses real limitations of either approach alone, but it also introduces complexity in understanding which retrieval mode is responsible for results, managing consistency between paths, and tuning the fusion logic. The source attribution DoorDash mentions helps with observability, but the dual-path architecture increases debugging surface area and requires careful thinking about failure modes.

The batch refresh model, while enabling offline LLM use, means personalization lags consumer behavior by the refresh interval. DoorDash acknowledges this by mentioning exploration of faster refresh and incremental updates, but the fundamental constraint remains that per-consumer generated content is computed on a schedule rather than reflecting real-time state. For use cases where immediate responsiveness to recent behavior is critical, this may be limiting.

The cost structure warrants examination. While DoorDash successfully amortizes LLM costs across refresh intervals and eligible cohorts, they’re still running batch LLM inference for millions of consumers, embedding all generated intents, maintaining large-scale vector indexes, and operating the hybrid retrieval infrastructure. The framework shifts costs from request-time to batch-time and changes the cost profile, but total system cost may still be substantial. Organizations should carefully model the economics based on their cohort sizes, refresh requirements, and retrieval volumes.

Overall, DoorDash’s framework represents mature, production-grade LLMOps engineering that successfully addresses the real constraint of deploying LLMs for personalization at scale—avoiding inline latency while maintaining meaningful per-user adaptation. The architectural principles are sound and the engineering choices show evidence of learning from production experience. Organizations should view this as a reference architecture for offline generative personalization, recognizing both its sophistication and the engineering investment required to implement it successfully.