## Overview This case study is drawn from a podcast conversation with Barak Turovsky, an executive in residence at Scale Venture Partners who previously served as a product leader at Google. He was instrumental in launching Google Translate using deep neural networks in 2015-2016, which he describes as the "first coming of AI." The discussion provides a practitioner's perspective on deploying LLMs in production, with lessons learned from both that early era and the current "second coming of AI" driven by ChatGPT and generative AI democratization. The core contribution of this discussion is a framework for evaluating which LLM use cases are viable for production deployment versus those that will require years of additional development. This framework is particularly valuable for MLOps and LLMOps practitioners trying to prioritize where to invest their engineering efforts. ## Historical Context: Google Translate as the First Large-Scale LLM Deployment Turovsky provides valuable historical context about the production challenges faced when deploying what we would now call transformer-based models at scale. When Google Brain researchers first approached the Translate team about using deep neural networks, the academic models had only been tested on datasets of around 10,000 sentences. Google Translate's production system, in contrast, operated on single-digit billions of training sentences for well-supported languages like Portuguese. The initial deep learning approach was approximately 100x slower than the existing statistical machine translation system in production. This latency gap was so severe that Google invested $130 million upfront to develop custom hardware (Tensor Processing Units/TPUs) without any clear monetization path guaranteed. This decision illustrates the scale of infrastructure investment that may be required to productionize AI at scale. The team launched 20 languages in just nine months, significantly faster than the initial three-year estimate, driven by engineering excitement about the transformative nature of the technology. However, even then they had to overcome hallucinations, latency issues, and quality concerns—the same challenges that plague LLM deployments today. ## The Accuracy vs. Fluency Framework Turovsky's central contribution to LLMOps thinking is his two-dimensional framework for evaluating use cases: **Axis 1: Accuracy Requirements** - Low accuracy tolerance: Use cases where factual correctness is critical (search, decision support, financial advice) - High accuracy tolerance: Use cases where creative output matters more than strict factual accuracy (poetry, fiction, brainstorming) **Axis 2: Fluency Requirements** - High fluency needs: Use cases requiring polished, eloquent, human-like text generation - Low fluency needs: Use cases where rough output is acceptable **Overlay: Stakes Level (color-coded)** - Green (low stakes): Creative tasks like writing poems, fiction, music lyrics where hallucination might even be a feature - Yellow (medium stakes): Workplace productivity tasks like drafting emails or presentations where humans can verify before sending - Red (high stakes): Search, booking systems, financial decisions where errors have real consequences and human verification at scale is impossible The key insight is that LLMs are currently much better suited for the green and yellow quadrants. The framework explicitly warns against pursuing red quadrant use cases prematurely, as the gap between demo and production is enormous. ## Production Challenges and Mitigations ### Hallucination Management Turovsky emphasizes that hallucinations are inherent to LLMs and will continue to exist. The mitigation strategy depends on the use case: - For creative use cases, hallucinations may be acceptable or even beneficial - For workplace productivity, the division of labor between human and machine can accommodate hallucinations—the machine handles "fluency" (eloquent writing) while humans verify accuracy - For high-stakes use cases, you may need supporting ML systems specifically designed to detect and filter hallucinations The team at Google Translate built auxiliary ML systems specifically to handle hallucinations even in the 2015-2016 timeframe. ### Latency and Cost Considerations The framework naturally aligns use cases with their latency tolerance: - Search requires instant responses, high accuracy, and very fresh results—making it extremely challenging for LLMs - Email drafting can tolerate waiting even an hour for a response, dramatically relaxing infrastructure requirements - Creative tasks similarly have relaxed latency requirements Cost scales with model size, which often correlates with accuracy. This creates a natural tension: high-accuracy use cases may require larger, more expensive models while simultaneously demanding lower latency. ### The Human Verification Scale Problem A critical insight for LLMOps practitioners: some use cases fundamentally cannot have human verification at scale. Search is the prime example—you cannot put a human behind every query to validate accuracy. This constraint alone may push certain use cases years into the future regardless of model improvements. In contrast, yellow-quadrant use cases like email drafting can achieve 70% productivity gains even with imperfect accuracy because the human verification step is already built into the workflow. ## User Feedback and Data Flywheel Turovsky emphasizes that the real competitive moat in LLM applications comes from collecting user feedback integrated into the workflow. This could take multiple forms: - Observing user adjustments to generated content - Explicit accept/reject signals - Implicit signals from whether users modify suggestions This aligns with modern LLMOps best practices around RLHF and continuous learning from user interactions. The example given of Grammarly-style interfaces where users correct suggestions demonstrates how feedback can be collected without interrupting the user experience. ## Open vs. Proprietary Models: Infrastructure Implications The discussion acknowledges uncertainty about whether the future will be dominated by proprietary models (OpenAI, Google) or custom models fine-tuned from open-source foundations. If the industry moves toward custom models, Turovsky predicts significant growth in: - Vector database infrastructure - Embedding and vectorization pipelines - Data preparation and preprocessing tools - Enterprise search engine integration This has direct implications for MLOps tooling investments. The observation that "Google doesn't have a moat, neither does OpenAI" suggests that differentiation will come from data and fine-tuning rather than base model access. ## Industry Impact Predictions Turovsky predicts four major areas of disruption: **Entertainment**: Already happening as the industry experiments aggressively with new technology (as they historically have with every major tech shift). This includes both positive applications and concerns around deepfakes and rights management. **Customer Interactions (Massive Cross-Industry Impact)**: Any company with 1+ million customers should expect to make internal knowledge bases accessible via LLMs across all channels (email, chat, voice). This could reduce costs by 40-50% while improving customer experience through: - Better intent understanding - Personalized rephrasing of technical content for non-technical users - Emotional intelligence in customer interactions (GPT-4's ability to detect and respond to user emotions) The warning here is that this is not as simple as "adding ChatGPT on top of Twilio." Companies will need: - Different tooling for hallucination detection - New skill sets for data cleanup and processing - Higher-level human agents to handle exceptions and high-stakes decisions - Completely re-engineered processes **Coding**: Tools like Copilot represent the beginning of a major shift in developer productivity. **Education**: GPT-4's performance on standardized tests is already impacting companies like Chegg and Pearson. This disruption is just beginning. ## Skills Evolution for ML Engineers The discussion concludes with advice for ML engineers working with product teams: **Technical Side**: - Stay current on model leaderboards comparing proprietary and open-source models - Understand embedding and vector database fundamentals - Learn how LLMs predict next tokens to understand data preparation requirements **Product Side**: - Challenge product managers on use cases - Distinguish between demos (working for 5% of users) and production-ready applications - Ask questions about user needs rather than accepting hype-driven requirements The emphasis is on ML engineers becoming more "LLM-friendly" with deeper understanding of both the capabilities and limitations of these systems, positioning themselves to contribute to product decisions rather than just technical implementation. ## Cautionary Notes Throughout the discussion, Turovsky provides balanced perspective that tempers AI hype: - The gap between demo and production is enormous, often requiring 80-90% of effort for the last 5% of use cases - Voice assistants (Alexa, Google Assistant) demonstrate that users don't adopt high-stakes use cases even when technology is ready—adoption of voice commerce remains minimal despite Amazon having perfect predictive capabilities - User trust builds slowly; high-stakes use cases may require 3-5 years of building trust through lower-stakes applications first - The Pareto principle applies: reducing hallucinations from 90% to 95% accuracy might take 10 years, and that still might not be enough for certain applications This framework provides a practical lens for LLMOps practitioners to evaluate where to invest their engineering efforts, prioritizing achievable use cases over aspirational ones.