## Overview This case study is drawn from a conference talk by a speaker with significant industry experience, having co-founded the messaging apps team at Apple before transitioning to work with LLMs starting in early 2023. The speaker subsequently joined LlamaIndex, where they created LlamaIndex TS (TypeScript version) and led partnerships. The talk distills practical lessons learned from building and deploying LLM applications in production environments, offering a candid and somewhat humorous look at the challenges practitioners face. The presentation is structured around "five things you might want to know before putting LLMs in production," providing actionable insights that go beyond theoretical concepts to address real-world pain points. It's worth noting that this is a practitioner-focused talk rather than a formal case study with quantified business results, so the lessons are primarily experiential rather than data-driven. ## Lesson 1: Recognizing When You're Actually in Production The speaker opens with an unconventional but memorable heuristic for determining if an LLM application is truly in production: whether users have attempted to input inappropriate or unexpected content. The example given is users typing offensive or irrelevant queries into chatbots. This leads to a critical point about **query classification** as a first line of defense in production systems. The key recommendation is that regardless of how sophisticated the underlying system is—whether it's a complex multi-agent architecture with planning capabilities, code generation, or image analysis—production LLM applications need robust input filtering and classification. The speaker emphasizes that there are "lots and lots of questions that your users might want to ask you that you just don't want to answer." Rather than attempting to handle all possible inputs with complex agent logic, the pragmatic approach is to implement query classification that can identify and reject problematic queries with a simple "I can't answer that" response. This insight speaks to a broader LLMOps principle: production systems need explicit boundaries and guardrails that protect both the system and the users from unintended behaviors. The complexity of multi-agent systems shouldn't be applied to every input; instead, a classification layer should route or filter queries before they enter the more expensive and potentially problematic reasoning chains. ## Lesson 2: Model-Specific Prompt Formatting The second lesson addresses the often-overlooked reality that different LLMs expect prompts formatted in different ways. The speaker notes that LLMs "talk in different languages" including JSON, Markdown, YAML, and surprisingly, XML. This last format is particularly relevant for Anthropic's Claude models, which work best with XML-structured prompts. The speaker illustrates this point with examples from Meta's Llama family: - **Llama 2** had a specific prompt format documented in Meta's official documentation - **Llama 3** introduced a completely different format that "looks nothing alike" - Llama 3's function calling implementation requires telling the model to respond in JSON format explicitly, even though function calling is already a structured interaction pattern The speaker points out a telling detail: Meta's official documentation for Llama 3 function calling contained a typo, suggesting that function calling "was not top of mind for the researchers when they launched this thing." This observation highlights a common LLMOps challenge—documentation quality and feature maturity vary significantly across providers and even across different capabilities within the same model family. The practical implication is that LLMOps teams need to invest time in understanding the specific prompting conventions for each model they use. What works for OpenAI's GPT-4 may not work optimally for Claude or Llama models, and even different versions of the same model family (Llama 2 vs Llama 3) can have substantially different requirements. ## Lesson 3: Small Changes, Big Consequences Perhaps the most technically specific lesson concerns how seemingly minor implementation changes can have outsized effects on system behavior. The speaker shares a concrete example involving OpenAI's embedding API: - The official OpenAI library provides an `embeddings.create` method - OpenAI's cookbook recommends using a utility function called `get_embedding` from `embedding_utils` - This utility function replaces newlines with spaces before creating embeddings - This preprocessing step was originally necessary due to a bug in earlier embedding models The speaker references Boris Power from OpenAI, who confirmed that newer embedding models no longer have the newline issue. However, when the speaker attempted to remove this preprocessing step from LlamaIndex (since it was no longer necessary), it immediately broke their most basic demo. This example underscores several important LLMOps principles: - **Legacy workarounds persist**: Even when the underlying issue is fixed, dependent systems may have adapted to the workaround in ways that create new dependencies - **Seemingly trivial changes matter**: Text preprocessing decisions like handling whitespace, punctuation, and special characters can significantly impact embedding quality and downstream retrieval performance - **Testing at the integration level is essential**: Unit-level changes that seem safe can break end-to-end functionality in unexpected ways ## Lesson 4: Function Calling Accuracy Limitations The fourth lesson provides a sobering assessment of the current state of function calling (tool use) in LLMs, which underpins many agent-based architectures. The speaker cites Mistral's testing when they launched Mistral Large, which showed that even the best models achieved only around 50% accuracy on function calling tasks. The speaker notes that Mistral's benchmarks showed their model "almost got to 50% accuracy," positioning themselves as better than GPT-4 and Claude. However, the practical implication is stark: "50% of the times is correct" is not a compelling reliability story for production applications. Additional concerns are raised about GPT-4 and GPT-4o Mini specifically: - GPT-4 has known issues with generating valid JSON for function calling - GPT-4o Mini reportedly has even worse JSON conformance - These are "commonly known problems" in the practitioner community The recommendation is clear: function calling capabilities are "still very early." LLMOps teams should be cautious about relying heavily on agent-based architectures that depend on reliable function calling, and should implement robust error handling and validation for function call outputs. ## Lesson 5: Data Quality and Document Parsing The final and perhaps most emphasized lesson is the importance of inspecting data quality, particularly when using document parsing for RAG (Retrieval-Augmented Generation) systems. The speaker references guidance from the AI Engineer conference: "always look at your data." A practical example is provided where the speaker and a colleague ran the same PDF (their slide deck) through two different document parsers: - One parser produced reasonable, readable text output - The other parser produced garbled output with strange character spacing (e.g., "I space a space back" instead of coherent text) The speaker emphasizes that no amount of sophisticated multi-agent RAG architecture can compensate for garbage input data. If the underlying parsed text is corrupted or poorly extracted, the entire system will fail regardless of how well-designed the retrieval and generation components are. This connects to a fundamental LLMOps principle: **data quality is upstream of model quality**. Teams often focus on prompt engineering, model selection, and architecture design while underinvesting in the data pipeline. Document parsing is particularly treacherous because: - Different parsers have different strengths and failure modes - PDFs are notoriously difficult to parse consistently - Visual inspection of parser output is necessary before trusting automated pipelines ## Broader Implications for LLMOps Practice While this talk doesn't provide quantified business outcomes or detailed technical architectures, it offers valuable practitioner wisdom that reflects the current maturity level of LLM deployments. Several themes emerge: **Defensibility over capability**: Rather than maximizing what an LLM system can do, production deployments often benefit from clearly defining what the system should *not* do and implementing robust boundaries. **Model heterogeneity is real**: The ecosystem includes models from OpenAI, Anthropic, Meta, Mistral, and others, each with different prompting conventions, strengths, and quirks. LLMOps practices need to accommodate this diversity. **Reliability lags capability**: Features like function calling are exciting but not yet reliable enough for many production use cases. Setting appropriate expectations and implementing fallbacks is essential. **Data quality requires vigilance**: Even with sophisticated tooling, fundamental data quality issues can undermine entire systems. Regular inspection and validation of data pipelines remains necessary. The speaker's background—transitioning from traditional software engineering at Apple to LLM development—also reflects a broader industry trend of experienced engineers discovering that many established software engineering practices need adaptation for the probabilistic, often unpredictable nature of LLM-based systems.