## Summary Buzzfeed, the digital media company known for its viral content, quizzes, and the Tasty food brand, embarked on an accelerated adoption of generative AI tools starting in late 2022. This case study from their engineering team details how they approached building production-ready LLM-powered applications, specifically chatbots and ChatGPT Plugins. The article provides a candid look at their technical journey, including dead ends and lessons learned, making it a valuable resource for understanding real-world LLMOps challenges in a media company context. The primary use cases driving their work included chatbots that allow users to engage with brands like Tasty in a conversational manner (described as consulting "a trusted homecook friend") and ChatGPT Plugins that create new distribution channels for Buzzfeed's content. The engineering team faced two fundamental challenges common to early LLM adopters: the dataset recency limitation (OpenAI's models at the time had a knowledge cutoff of October 2021) and the limited context window (4,000 tokens for GPT-3.5). ## Early Experimentation Phase Buzzfeed's initial products, including games like "Under The Influencer" and the "Infinity Quizzes," were built using vanilla ChatGPT v3.5 with carefully crafted prompts. This represents the classic prompt engineering approach that many organizations started with. The team discovered several limitations during this phase that offer useful lessons for LLMOps practitioners. The model exhibited charming but harmless quirks like the inability to perform basic arithmetic, but more significant issues emerged around complex prompts. The team found that performance declined notably when system prompts contained five or more rules. Their solution was to break complex instructions into multiple API calls that informed the control flow of the application. This pattern of decomposing complex tasks into simpler subtasks is a common technique in production LLM systems and represents an early form of the "chain of thought" reasoning that would later become formalized in frameworks. ## The Self-Hosting Dead End In an interesting aside, the team explored self-hosting fine-tuned LLMs like FLAN-T5, motivated in part by innovations like LoRA (Low-Rank Adaptation) that made fine-tuning on proprietary data more feasible. However, they ultimately abandoned this approach due to economic considerations. The article notes that self-hosting LLMs is "a game of thin margins that can shrink dramatically whenever OpenAI makes an announcement." This reflects a common dilemma in LLMOps: the build-vs-buy decision for model infrastructure, where the rapid pace of improvement and price reductions from commercial providers can undermine the economics of self-hosted solutions. ## The RAG Solution: BFN Answering Machine The breakthrough came with an internal prototype called "BFN Answering Machine," a chatbot with knowledge of recent Buzzfeed News articles. This system addressed the dataset recency limitation using a semantic search combined with LLMs approach, which the industry would come to call Retrieval-Augmented Generation (RAG). The technical implementation involved several key components. First, they generated embeddings for their corpus of Buzzfeed News articles, projecting the content into vector space. They applied the same transformation to user queries and used nmslib for nearest neighbor search to find matching articles. The top k matches were then included in the prompt passed to OpenAI's Completion API, allowing the system to authoritatively answer questions about current events that occurred after OpenAI's training data cutoff. This approach elegantly solved the recency problem by keeping the LLM's context grounded in up-to-date content from Buzzfeed's own corpus, rather than relying solely on the model's parametric knowledge. ## Production Architecture Evolution While the nmslib-based prototype worked in a Colab Notebook, it wasn't production-ready. The team invested in improving their Nearest Neighbor Search Architecture, which had already been powering personalization and content recommendations for years. This is an important point: they leveraged existing infrastructure investments rather than building everything from scratch. The previous architecture used Google's Matching Engine and internal microservices for embedding content (recipes, articles) and hydrating search results back into a standardized format. The new architecture introduced several significant improvements. They moved to an event-driven system powered by NSQ, a core messaging technology in Buzzfeed's stack. They added batch endpoints to simplify experimentation, allowing data scientists to build new large indexes from notebooks while also improving throughput for production workloads. A key infrastructure decision was migrating from Google's Matching Engine to Pinecone as their vector database backend. The decoupled interface for backend vector databases made this migration possible and resulted in immediate cost savings on their GCP bill. This modular architecture design is a best practice in LLMOps, allowing teams to swap components as the vendor landscape evolves. ## The LangChain Experiment and Abandonment To address the limited context window challenge, the team explored retrieval-augmented generation using LangChain, a popular framework that formalizes control flow logic powered by LLM responses to simpler subqueries. LangChain offered an out-of-the-box implementation of ReAct (Reasoning and Acting), a technique that synergizes reasoning and action by having LLMs generate both reasoning traces and task-specific actions in an interleaved manner. While they appreciated LangChain's concise representation and ease of getting started, the team eventually outgrew its "batteries included" philosophy. They cite several specific pain points that are instructive for other teams considering similar frameworks. They needed more control over error handling and instrumentation with their existing metrics tooling, particularly for tracking how many calls were being made to OpenAI. They needed finer control over when calls were made to OpenAI, likely for cost management and latency optimization. Perhaps most frustratingly, LangChain crashed when learned heuristics (such as which tool to use in which scenario) conflicted with provided system prompt instructions. The decision to abandon LangChain in favor of a custom implementation aligns with broader industry sentiment at the time. The article notes they were "like many others" in this decision, linking to a Hacker News discussion on the topic. This reflects a tension in LLMOps between the convenience of high-level abstractions and the need for production-grade control over system behavior. ## Native ReAct Implementation The team's solution was to build what they call "native ReAct" because of its faithfulness to the original research paper. This approach handles reasoning and candidate generation internally while still leveraging OpenAI models for actual text generation. The architecture they describe shows a chatbot using retrieval-augmented generation with clear separation of concerns. This custom implementation gave them the control they needed while maintaining interoperability with their existing monitoring and metrics reporting stack. The ability to integrate with existing observability infrastructure is crucial for production LLM systems, enabling teams to track costs, debug issues, and understand system behavior. ## Key LLMOps Lessons Several important lessons emerge from Buzzfeed's experience. The economics of LLM infrastructure are volatile, and decisions around self-hosting need to account for rapid changes in commercial pricing and capabilities. RAG architectures can effectively extend LLM knowledge beyond training data cutoffs by grounding responses in up-to-date proprietary content. Existing infrastructure investments, like Buzzfeed's Nearest Neighbor Search Architecture, can be leveraged for new LLM use cases. Framework abstractions like LangChain may need to be replaced with custom implementations as production requirements become clearer. Instrumentation and monitoring integration should be a first-class concern when building LLM applications. The case study also implicitly highlights the importance of iterative development. The team moved from simple prompt engineering to exploring self-hosted models to prototyping with notebooks to building production-ready infrastructure, learning and adapting at each stage. This experimental approach, combined with willingness to abandon approaches that don't work (like FLAN-T5 self-hosting and LangChain), represents a mature engineering culture well-suited to the rapidly evolving LLMOps landscape.