## Summary Airbnb's engineering team published this case study in late 2022 describing their adoption of text generation models to improve customer support operations at scale. The company transitioned from traditional discriminative ML models (classifiers) to generative models based on large-scale language models. This shift addressed three fundamental challenges in their customer support products: handling long-tail corner cases, scaling operations cost-effectively, and reducing the expensive burden of labeling training data. The case study presents three distinct production use cases: content recommendation for the Help Center, real-time agent assistance through a question-answering model, and chatbot paraphrasing to improve user engagement. Each use case demonstrates different aspects of operationalizing text generation models in a high-traffic production environment serving millions of users. ## Key Principles Behind Their Approach Airbnb identified several beneficial traits of text generation models that guided their implementation strategy: **Knowledge Encoding**: The team leveraged years of accumulated records from human agents helping guests and hosts. Rather than treating the model as a simple transformation function from input to output, they focused on encoding domain knowledge through large-scale pre-training and transfer learning. This fundamentally changed their approach from feature engineering and label preparation to knowledge encoding and prompt design. The ability to encode "knowledge about solving users' travel problems" directly into the models proved significantly more effective than traditional classification approaches. **Unsupervised Learning Benefits**: One of the most operationally significant advantages was the unsupervised nature of text generation models. Traditional approaches required extensive manual labeling, which was both costly and challenging due to the need to define comprehensive label taxonomies for user issues and intents. Real-world customer support problems have long-tail distributions with nuanced corner cases that don't scale well with human-driven labeling efforts. By using generative models, the pre-training process forces the model to implicitly learn the problem taxonomy, effectively doing some of the data labeling design internally. **Prompt-Based Unification**: The team adopted prompt engineering as a central technique, viewing all ML tasks as different manifestations of a single language modeling problem. They designed prompts as textual instructions that inform the model of the task and set expectations for output format and content. Natural language annotations and hints are incorporated to further contextualize the problem. ## Use Case 1: Content Recommendation Model The content recommendation system powers both Airbnb's Help Center search and support content recommendations in their Helpbot. The system uses pointwise ranking to determine document ordering for users. **Previous Architecture**: Prior to 2022, they used XLMRoBERTa as an encoder-only architecture with an arbitrary classification head. The ranker took textual representations of the user's issue description and candidate documents (title, summary, keywords) to compute relevance scores. **New Architecture**: They transformed the binary classification problem into a prompt-based language generation problem using the MT5 model with encoder-decoder architecture. The key innovation was prepending a prompt to the issue description that explicitly asks for a binary "Yes" or "No" answer about whether a document would be helpful. Annotations provide hints about the intended roles of various input text parts. **Personalization**: They expanded the issue description input with textual representations of user information and reservation details to enable personalization. **Evaluation and Results**: Using production traffic data sampled from the same distribution as training data, the generative model showed significant improvements in the key performance metric for support document ranking. Online A/B experiments in the Help Center with millions of active users confirmed that the generative model recommends documents with significantly higher relevance compared to the classification-based baseline. ## Use Case 2: Real-Time Agent Assistant Model This system provides just-in-time guidance to customer support agents, helping them resolve user issues efficiently by suggesting relevant templates during conversations. **Problem Formulation**: The model needs to answer granular intent questions about user messages, such as: "Is this message about a cancellation?", "What cancellation reason did this user mention?", "Is this user canceling due to COVID sickness?", "Did this user accidentally book a reservation?" **Architecture**: They developed a "mastermind" Question-Answering model using the generative architecture. The input consists of concatenated multi-round user-agent conversation history, with prompts designed to capture specific intents at serving time. Template recommendations are gated by a combination of API checks and model intent checks to ensure suggestions comply with CS policy. **Training Data Strategy**: They experimented with various training dataset compositions including annotation-based data and logging-based data with post-processing. Annotation datasets typically have higher precision but lower coverage with more consistent noise patterns. Logging datasets have lower precision but higher case coverage with more random noise. The best performance came from combining both dataset types. **Infrastructure for Training**: Due to the large model parameter sizes, they leveraged Microsoft's DeepSpeed library for multi-GPU training. This reduced training time from weeks to days. However, hyperparameter tuning still required significant experimentation, which they conducted on smaller datasets to establish parameter direction before full-scale training. **Model Backbones Tested**: They experimented with t5-base and Narrativa models. **Production Results**: Online testing with real CS ambassadors showed large engagement rate improvements, indicating the model successfully assists agents in problem resolution. ## Use Case 3: Chatbot Paraphrasing Model This system addresses a crucial UX challenge: users often disengage from chatbots regardless of the underlying model quality because they doubt whether the bot truly understands their problem. **Solution Approach**: The paraphrasing model rephrases the user's problem description back to them, building confidence that the bot understands correctly before proceeding. This mirrors a common human agent technique using phrases like "I understand that you..." **Training Data Generation**: They leveraged years of agent-user communication data by extracting conversations where agent replies start with "I understand that you..." This simple heuristic provided millions of training labels without manual labeling effort—a key example of their unsupervised learning philosophy in practice. **Model Selection**: They tested popular sequence-to-sequence transformer backbones including BART, PEGASUS, and T5, as well as autoregressive models like GPT2. For their specific use case, the T5 model produced the best performance. **Key Challenge - Generic Outputs**: A major challenge was that the model tended to generate bland, generic, uninformative replies. For example, the model would output "I understand that you have some issues with your reservation" for many different inputs—technically correct but too generic to be useful. **Solutions Attempted**: They tried several approaches including building a backward model to predict P(Source|target) for reranking to filter generic results, and rule-based or model-based filters. **Winning Solution - Training Data Curation**: The most effective solution involved tuning the training data itself. They used Sentence-Transformers to run text clustering on training target data based on pre-trained similarity models. Analysis revealed that training data contained too many generic meaningless replies, which caused the model to replicate this behavior. They labeled generic clusters and used Sentence-Transformers to filter them from training data, resulting in a high-quality production model. ## LLMOps Considerations and Lessons **Data Quality Over Quantity**: The chatbot paraphrasing case particularly demonstrates that data quality matters more than raw quantity. Filtering training data based on semantic clustering significantly improved output quality. **Hybrid Data Strategies**: Combining annotation-based and logging-based data provided optimal results for the agent assistant model, suggesting that production systems benefit from mixing high-quality labeled data with broader but noisier production logs. **Prompt Design as First-Class Concern**: Across all use cases, prompt engineering proved essential. Prompts were naturally aligned with human annotation questions, making them interpretable and maintainable. **Evaluation Pipeline**: The team maintained rigorous evaluation practices, using production traffic data matching training distributions for offline evaluation and conducting online A/B experiments with real users before full deployment. **Infrastructure Investments**: The need for DeepSpeed to handle large model training suggests significant infrastructure investment was required. The reduction from weeks to days of training time has major implications for iteration speed and experimentation velocity. **Incremental Rollout**: The case study mentions A/B testing and experimentation with real CS ambassadors, indicating a measured approach to production deployment rather than wholesale replacement of existing systems. It's worth noting that while the case study reports positive results across all three use cases, specific quantitative metrics beyond qualitative statements like "significant improvements" and "large engagement rate improvement" are not provided. This is common in industry publications but makes it difficult to independently assess the magnitude of improvements. The fundamental shift from discriminative to generative models appears to have delivered real value, but readers should recognize that the source is Airbnb's own engineering blog, which naturally emphasizes successes.