Target's marketing teams needed to accurately forecast campaign performance before launch to optimize budget allocation and improve guest experiences. The company rebuilt their campaign similarity matching system using a RAG architecture that combines semantic embeddings, retrieval from historical campaign data, and LLM-based filtering and ranking. The new system achieved 100% coverage with top-3 recommendations (compared to 75% with top-1), eliminating manual intervention while providing explainable, auditable matches that help predict campaign outcomes and improve offer redemption models.
Target developed and deployed a production LLM-powered system to improve marketing campaign performance forecasting through semantic retrieval and RAG architecture. The use case addresses a critical business problem: marketing teams must decide campaign investment levels before launch, and these decisions directly impact both campaign effectiveness and budget efficiency. The system identifies and ranks historically similar campaigns to predict how new campaigns will perform, enabling more accurate offer-redemption models and ultimately delivering more relevant, personalized offers to customers while optimizing marketing spend.
This case study represents a rebuild of an existing rule-based system. The original approach relied on basic embeddings and rule-based logic that worked adequately in simpler scenarios but degraded as Target’s marketing ecosystem evolved toward more sophisticated and niche campaign types. The previous system suffered from increased false positive rates and required significant manual intervention, limiting scalability and automation potential. Beyond accuracy improvements, stakeholders needed transparency and the ability to tune the system, which the new GenAI-based solution provides through ranked results with clear rationales.
The production system follows a multi-stage RAG architecture with several key components working together. The overall workflow treats each incoming campaign request as a query, with the campaign described through markdown text, guest intent classifications (such as “engaged” or “lapsing”), pyramid and category metadata, and other structured attributes. Historical campaigns and offer data are aggregated, cleaned, and embedded using language models, then indexed in Target’s proprietary AI grounding service.
When a new campaign needs forecasting, the system embeds the campaign’s metadata and retrieves the most similar historical campaigns from the index. For example, if a new campaign aims to drive trips among lower-engaged guests in the apparel category, the retrieval system automatically surfaces past clothing promotions with identical objectives. The retrieved candidates then pass through an LLM-based filter and ranker that applies a structured prompt encoding Target’s matching criteria hierarchy to produce the final ranked set of comparable campaigns.
The architecture consists of three primary stages: (A) grounding data preparation, (B) retrieval strategy, and (C) LLM-based filtering and ranking. This separation of concerns allows each component to be optimized independently while maintaining end-to-end system performance.
The team invested heavily in data preparation, recognizing that accurate matching depends not just on advanced models but on well-prepared, unified data. Campaign and offer data typically arrives in unstructured or semi-structured formats, requiring significant cleaning, connecting, and enrichment. The data pipeline unifies, filters, and engineers features to produce a single consistent “final offer pool” used for grounding the retrieval system.
Several filters keep the dataset focused and relevant. The system limits to non-geo-segmented, omni-channel digital promotions that are standard and complete. It focuses specifically on basket-based offers with dollar-off or percent-off rewards while excluding manufacturer coupons and rebates. This filtering ensures the grounding data represents comparable campaign types rather than introducing noise from fundamentally different promotion mechanisms.
Beyond filtering, the team engineered new features that add performance context and metadata. These include metrics like distribution volume, redemption counts, and total discount spend. The pipeline also derives governance and rollout indicators such as whether a campaign was sitewide or targeted, mass-market or niche, and whether it was “fully scaled” (a signal inferred from campaign text). This enriched dataset feeds directly into the retrieval index within Target’s AI platform, providing the foundation for semantic search.
In the candidate generation stage, the team evaluated multiple retrieval configurations through offline evaluation, testing different prompt variations and index strategies to identify the most effective setup. The evaluation revealed that a multi-index approach delivered the best raw retrieval quality compared to single-index alternatives.
The multi-index strategy provides several advantages beyond raw accuracy. It produces more explainable results by allowing different indices to capture different aspects of campaign similarity. It improved consistency across niche campaign types that might be underrepresented in a single unified index. Perhaps most importantly from an LLMOps perspective, it provides a governance-ready framework for future policy and safety controls, allowing different indices to enforce different business rules or constraints.
While the text doesn’t specify the exact nature of the multiple indices, the approach likely separates campaigns by major dimensions such as product category, audience segment, offer mechanics, or temporal characteristics. This separation allows the retrieval system to apply domain-specific embeddings or matching logic before combining results.
The LLM prompt represents a critical component of the system’s effectiveness and is designed to provide both context and constraints for consistent, explainable retrieval. Each prompt carries three key components that work together to guide the model’s reasoning: input campaign facts providing structured metadata about the campaign being analyzed, retrieved candidates (top N) representing similar past campaigns returned by the retrieval engine, and a matching criteria hierarchy providing an ordered set of attributes used to guide comparison and ranking.
The matching criteria hierarchy is particularly important for LLMOps practices. By explicitly encoding domain knowledge about which attributes matter most for campaign similarity (such as product domain, audience segment, and timing), the prompt helps anchor the LLM’s reasoning in relevant business logic rather than relying purely on the model’s latent understanding. This structured approach improves consistency and makes the system’s decisions more auditable and explainable to business stakeholders.
The team tested multiple candidate pool sizes and re-ranker cutoffs to optimize the precision-recall tradeoff. Precision helps ensure the relevance of retrieved campaigns, while recall reduces the risk of overlooking valuable historical campaigns. Through experimentation, they found optimal performance by retrieving a baseline pool of 10 candidates and having the LLM select the top 3 (K=3). This configuration achieved high precision without requiring manual intervention.
The evaluation strategy employed a time-separated train-test setup on a diverse set of recent marketing campaigns, which is appropriate for temporal forecasting tasks where future campaigns must be predicted from past data. The key performance measure was coverage rate, defined as how often the system provided at least one acceptable recommendation without requiring human override.
The results demonstrate significant improvement over the previous system. With only the top recommendation (K=1), the new system achieved 75% coverage, successfully providing suitable matches for most campaigns. This already represents strong performance for an automated system. However, expanding to the top three matches (K=3) achieved 100% coverage, meaning every campaign contained at least one strong and contextually aligned recommendation. This effectively eliminates the need for manual search and correction that plagued the previous rule-based system.
Beyond coverage metrics, the team notes that the improved matching grounds the offer propensity model with more contextually relevant training data, resulting in improved forecast accuracy and campaign performance. However, the case study doesn’t provide specific quantitative metrics on forecast accuracy improvement or downstream business impact such as redemption rate increases or ROI improvements. This omission makes it difficult to assess the full business value of the system beyond the coverage improvements.
Recognizing that large language models tend to hallucinate, the team introduced an additional validation layer to mitigate risk. While the text doesn’t detail the specific validation mechanisms, this represents important LLMOps practice for production systems. Possible validation approaches might include checking that the LLM’s output matches expected structural formats, verifying that cited campaign IDs exist in the grounding database, or ensuring that the rationales reference actual campaign attributes rather than invented details.
This validation layer is particularly important given that the system’s outputs directly influence marketing investment decisions. Hallucinated recommendations or spurious rationales could lead to poor forecasts and wasted budget. The explicit acknowledgment of hallucination risk and the implementation of mitigating controls demonstrates mature LLMOps thinking.
A key design goal beyond accuracy was making the system trustworthy and tunable for business stakeholders. The new system returns ranked results with clear rationales, making matches auditable and explainable. This explainability allows marketing teams to understand why certain historical campaigns were selected as comparable, enabling them to validate the system’s reasoning and refine matching criteria with data science teams.
The shift from opaque rule-based matching to explainable LLM-based ranking represents an interesting pattern in LLMOps. While traditional systems are often considered more interpretable than neural approaches, in practice, complex rule systems can become inscrutable as they grow. The LLM-based approach, when properly prompted to provide rationales, can actually deliver better explainability by articulating its reasoning in natural language that business users can evaluate.
While the case study doesn’t extensively detail deployment infrastructure, several production considerations emerge from the text. The system integrates with Target’s in-house AI grounding service for embedding storage and retrieval, suggesting integration with existing ML platform capabilities rather than standalone deployment. The multi-index approach and structured data pipeline indicate significant engineering investment in the data layer supporting the LLM application.
The system appears designed for batch or near-real-time processing rather than ultra-low-latency inference, which is appropriate for campaign forecasting use cases where decisions happen days or weeks before launch rather than in milliseconds. This allows for more sophisticated retrieval and reranking without latency constraints that would complicate production deployment.
The coverage metric (percentage of campaigns receiving acceptable recommendations without manual intervention) serves as a practical production monitoring metric that directly measures business value. This is more meaningful than abstract model performance metrics and provides clear signals when the system requires attention.
While the case study presents impressive coverage improvements, several aspects warrant balanced assessment. The evaluation focuses primarily on coverage rates and doesn’t provide detailed quantitative metrics on forecast accuracy improvement or downstream business impact such as redemption rate changes or marketing ROI. Without these metrics, it’s difficult to fully assess whether the increased coverage translates to material business value or simply automates existing decision quality.
The text mentions that the previous rule-based system suffered from “increased false positive rates” but doesn’t quantify false positive or false negative rates for the new system. Coverage at K=3 reaching 100% means at least one acceptable match was found among the top three recommendations, but doesn’t indicate how many of the three recommendations were actually acceptable. If the system returns one good match and two poor matches, that represents different quality than returning three equally good matches.
The multi-index approach and LLM reranking add significant complexity compared to simpler embedding-based retrieval. While the team conducted offline evaluation to justify this complexity, production systems must balance accuracy against operational complexity, maintenance burden, and cost. The case study doesn’t address computational costs, inference latency, or operational overhead of maintaining multiple indices and LLM inference infrastructure.
The team’s roadmap focuses on closing the loop by feeding real-world performance back into retrieval tuning, creating an adaptive system that learns dynamically from actual campaign outcomes rather than just historical patterns. This represents sophisticated LLMOps thinking about continuous learning and system evolution.
Implementing this feedback loop will require instrumentation to track predicted versus actual campaign performance, attribution mechanisms to connect forecasts to outcomes, and retraining or fine-tuning workflows to incorporate new learning. The distinction between updating retrieval indices with new campaigns versus actually tuning retrieval and ranking based on prediction accuracy represents different levels of sophistication in the adaptive learning vision.
This case study demonstrates several indicators of LLMOps maturity. The system rebuilds an existing production capability with clear business justification rather than pursuing AI for its own sake. The architecture separates concerns between data preparation, retrieval, and LLM-based ranking, allowing independent optimization and evolution. The team conducted rigorous offline evaluation before deployment, testing multiple configurations to optimize the precision-recall tradeoff.
The emphasis on explainability and auditability shows understanding that production systems must serve business users beyond just maximizing accuracy metrics. The explicit consideration of hallucination risks and implementation of validation layers demonstrates awareness of LLM-specific failure modes. The clear production metric (coverage rate) directly measures business value rather than abstract model performance.
However, some aspects suggest opportunities for further maturation. The lack of detailed quantitative metrics on business impact makes it difficult to assess ROI. The text doesn’t discuss monitoring, alerting, or operational procedures for the production system. There’s no mention of A/B testing or gradual rollout strategies to validate the new system against the previous approach in live traffic before full deployment.
Target’s marketing campaign forecasting system represents a substantive application of RAG architecture and LLMs to a real production problem with clear business value. The combination of semantic retrieval from carefully curated historical data with LLM-based filtering and ranking achieves strong coverage while providing the explainability and auditability that business stakeholders require. The system demonstrates thoughtful LLMOps practices in areas like prompt engineering, hallucination mitigation, and evaluation methodology, though some aspects like detailed business impact metrics and operational procedures remain underspecified in the public case study.
DoorDash faced the challenge of personalizing experiences across a massive, diverse catalog spanning restaurants, grocery, retail, and other local commerce categories for millions of users with rapidly shifting intents. Traditional collaborative filtering and deep learning approaches could not adapt quickly enough to short-lived, high-context moments like Black Friday or individual life events. DoorDash developed a hybrid architecture that leverages LLMs for product understanding, consumer profile generation in natural language, and content blueprint creation, while maintaining traditional deep learning models for efficient last-mile ranking and retrieval. This approach enables the platform to serve dynamic, moment-aware personalization that adapts to real-time user intent while managing latency and cost constraints. The system uses GEPA optimization within DSPy for compound AI system tuning, combines offline LLM processing with online signal blending, and evaluates performance through quantitative metrics, LLM-as-judge, and human feedback.
DoorDash addressed longstanding bottlenecks in search and recommendation quality across their food, grocery, retail, and gifting verticals by using LLMs to generate rich, standardized merchant and item profiles at scale, then encoding those profiles with off-the-shelf embedding models. Traditional behavioral embedding approaches failed to capture semantic nuances in transactional, intent-driven sessions with sparse engagement data, while pure content approaches suffered from poor metadata quality. By leveraging LLM-generated profiles combined with carefully selected embedding models (gemini-embedding-001 with 256-dimensional MRL), DoorDash achieved substantial improvements: semantic search reduced null search rates by 3.65% and increased CVR by 0.66%, while generative personalized carousels increased homepage order rate by 2.4% and offline precision improved from 68% to 85%. The content-first embedding strategy proved especially effective for cold-start scenarios, tail queries, and ensuring fairness to small merchants.
DoorDash's Core Consumer ML team developed a GenAI-powered context shopping engine to address the challenge of lost user intent during in-app searches for items like "fresh vegetarian sushi." The traditional search system struggled to preserve specific user context, leading to generic recommendations and decision fatigue. The team implemented a hybrid approach combining embedding-based retrieval (EBR) using FAISS with LLM-based reranking to balance speed and personalization. The solution achieved end-to-end latency of approximately six seconds with store page loads under two seconds, while significantly improving user satisfaction through dynamic, personalized item carousels that maintained user context and preferences. This hybrid architecture proved more practical than pure LLM or deep neural network approaches by optimizing for both performance and cost efficiency.
