BuzzFeed Tech tackled the challenges of integrating LLMs into production by addressing dataset recency limitations and context window constraints. They evolved from using vanilla ChatGPT with crafted prompts to implementing a sophisticated retrieval-augmented generation system. After exploring self-hosted models and LangChain, they developed a custom "native ReAct" implementation combined with an enhanced Nearest Neighbor Search Architecture using Pinecone, resulting in a more controlled, cost-efficient, and production-ready LLM system.
Buzzfeed, the digital media company known for its viral content, quizzes, and the Tasty food brand, embarked on an accelerated adoption of generative AI tools starting in late 2022. This case study from their engineering team details how they approached building production-ready LLM-powered applications, specifically chatbots and ChatGPT Plugins. The article provides a candid look at their technical journey, including dead ends and lessons learned, making it a valuable resource for understanding real-world LLMOps challenges in a media company context.
The primary use cases driving their work included chatbots that allow users to engage with brands like Tasty in a conversational manner (described as consulting “a trusted homecook friend”) and ChatGPT Plugins that create new distribution channels for Buzzfeed’s content. The engineering team faced two fundamental challenges common to early LLM adopters: the dataset recency limitation (OpenAI’s models at the time had a knowledge cutoff of October 2021) and the limited context window (4,000 tokens for GPT-3.5).
Buzzfeed’s initial products, including games like “Under The Influencer” and the “Infinity Quizzes,” were built using vanilla ChatGPT v3.5 with carefully crafted prompts. This represents the classic prompt engineering approach that many organizations started with. The team discovered several limitations during this phase that offer useful lessons for LLMOps practitioners.
The model exhibited charming but harmless quirks like the inability to perform basic arithmetic, but more significant issues emerged around complex prompts. The team found that performance declined notably when system prompts contained five or more rules. Their solution was to break complex instructions into multiple API calls that informed the control flow of the application. This pattern of decomposing complex tasks into simpler subtasks is a common technique in production LLM systems and represents an early form of the “chain of thought” reasoning that would later become formalized in frameworks.
In an interesting aside, the team explored self-hosting fine-tuned LLMs like FLAN-T5, motivated in part by innovations like LoRA (Low-Rank Adaptation) that made fine-tuning on proprietary data more feasible. However, they ultimately abandoned this approach due to economic considerations. The article notes that self-hosting LLMs is “a game of thin margins that can shrink dramatically whenever OpenAI makes an announcement.” This reflects a common dilemma in LLMOps: the build-vs-buy decision for model infrastructure, where the rapid pace of improvement and price reductions from commercial providers can undermine the economics of self-hosted solutions.
The breakthrough came with an internal prototype called “BFN Answering Machine,” a chatbot with knowledge of recent Buzzfeed News articles. This system addressed the dataset recency limitation using a semantic search combined with LLMs approach, which the industry would come to call Retrieval-Augmented Generation (RAG).
The technical implementation involved several key components. First, they generated embeddings for their corpus of Buzzfeed News articles, projecting the content into vector space. They applied the same transformation to user queries and used nmslib for nearest neighbor search to find matching articles. The top k matches were then included in the prompt passed to OpenAI’s Completion API, allowing the system to authoritatively answer questions about current events that occurred after OpenAI’s training data cutoff.
This approach elegantly solved the recency problem by keeping the LLM’s context grounded in up-to-date content from Buzzfeed’s own corpus, rather than relying solely on the model’s parametric knowledge.
While the nmslib-based prototype worked in a Colab Notebook, it wasn’t production-ready. The team invested in improving their Nearest Neighbor Search Architecture, which had already been powering personalization and content recommendations for years. This is an important point: they leveraged existing infrastructure investments rather than building everything from scratch.
The previous architecture used Google’s Matching Engine and internal microservices for embedding content (recipes, articles) and hydrating search results back into a standardized format. The new architecture introduced several significant improvements. They moved to an event-driven system powered by NSQ, a core messaging technology in Buzzfeed’s stack. They added batch endpoints to simplify experimentation, allowing data scientists to build new large indexes from notebooks while also improving throughput for production workloads.
A key infrastructure decision was migrating from Google’s Matching Engine to Pinecone as their vector database backend. The decoupled interface for backend vector databases made this migration possible and resulted in immediate cost savings on their GCP bill. This modular architecture design is a best practice in LLMOps, allowing teams to swap components as the vendor landscape evolves.
To address the limited context window challenge, the team explored retrieval-augmented generation using LangChain, a popular framework that formalizes control flow logic powered by LLM responses to simpler subqueries. LangChain offered an out-of-the-box implementation of ReAct (Reasoning and Acting), a technique that synergizes reasoning and action by having LLMs generate both reasoning traces and task-specific actions in an interleaved manner.
While they appreciated LangChain’s concise representation and ease of getting started, the team eventually outgrew its “batteries included” philosophy. They cite several specific pain points that are instructive for other teams considering similar frameworks. They needed more control over error handling and instrumentation with their existing metrics tooling, particularly for tracking how many calls were being made to OpenAI. They needed finer control over when calls were made to OpenAI, likely for cost management and latency optimization. Perhaps most frustratingly, LangChain crashed when learned heuristics (such as which tool to use in which scenario) conflicted with provided system prompt instructions.
The decision to abandon LangChain in favor of a custom implementation aligns with broader industry sentiment at the time. The article notes they were “like many others” in this decision, linking to a Hacker News discussion on the topic. This reflects a tension in LLMOps between the convenience of high-level abstractions and the need for production-grade control over system behavior.
The team’s solution was to build what they call “native ReAct” because of its faithfulness to the original research paper. This approach handles reasoning and candidate generation internally while still leveraging OpenAI models for actual text generation. The architecture they describe shows a chatbot using retrieval-augmented generation with clear separation of concerns.
This custom implementation gave them the control they needed while maintaining interoperability with their existing monitoring and metrics reporting stack. The ability to integrate with existing observability infrastructure is crucial for production LLM systems, enabling teams to track costs, debug issues, and understand system behavior.
Several important lessons emerge from Buzzfeed’s experience. The economics of LLM infrastructure are volatile, and decisions around self-hosting need to account for rapid changes in commercial pricing and capabilities. RAG architectures can effectively extend LLM knowledge beyond training data cutoffs by grounding responses in up-to-date proprietary content. Existing infrastructure investments, like Buzzfeed’s Nearest Neighbor Search Architecture, can be leveraged for new LLM use cases. Framework abstractions like LangChain may need to be replaced with custom implementations as production requirements become clearer. Instrumentation and monitoring integration should be a first-class concern when building LLM applications.
The case study also implicitly highlights the importance of iterative development. The team moved from simple prompt engineering to exploring self-hosted models to prototyping with notebooks to building production-ready infrastructure, learning and adapting at each stage. This experimental approach, combined with willingness to abandon approaches that don’t work (like FLAN-T5 self-hosting and LangChain), represents a mature engineering culture well-suited to the rapidly evolving LLMOps landscape.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
This panel discussion features three AI-native companies—Delphi (personal AI profiles), Seam AI (sales/marketing automation agents), and APIsec (API security testing)—discussing their journeys building production LLM systems over three years. The companies address infrastructure evolution from single-shot prompting to fully agentic systems, the shift toward serverless and scalable architectures, managing costs at scale (including burning through a trillion OpenAI tokens), balancing deterministic workflows with model autonomy, and measuring ROI through outcome-based metrics rather than traditional productivity gains. Key technical themes include moving away from opinionated architectures to let models reason autonomously, implementing state machines for high-confidence decisions, using tools like Pydantic AI and Logfire for instrumentation, and leveraging Pinecone for vector search at scale.
eBay developed Mercury, an internal agentic framework designed to scale LLM-powered recommendation experiences across its massive marketplace of over two billion active listings. The platform addresses the challenge of transforming vast amounts of unstructured data into personalized product recommendations by integrating Retrieval-Augmented Generation (RAG) with a custom Listing Matching Engine that bridges the gap between LLM-generated text outputs and eBay's dynamic inventory. Mercury enables rapid development through reusable, plug-and-play components following object-oriented design principles, while its near-real-time distributed queue-based execution platform handles cost and latency requirements at industrial scale. The system combines multiple retrieval mechanisms, semantic search using embedding models, anomaly detection, and personalized ranking to deliver contextually relevant shopping experiences to hundreds of millions of users.
