## Overview This case study presents Meta's comprehensive approach to building and scaling LLM inference infrastructure, as shared by Charlotte Qi, who works on LLM inference at Meta's AI infrastructure team. The presentation covers Meta's journey from 2023 onwards in developing production-grade LLM serving capabilities that power Meta AI, smart glasses, and extensive internal ML workflows including RLHF processing that handles hundreds of millions of examples during busy periods. Meta's approach demonstrates the evolution from simple model serving to building what essentially becomes a "distributed operating system" for LLMs. The case study illustrates the multifaceted challenges of productionizing large language models and the sophisticated engineering solutions required to achieve acceptable performance, cost efficiency, and reliability at Meta's scale. ## Technical Foundation and Model Runner Development Meta's LLM serving infrastructure begins with addressing the fundamental challenge of LLM inference patterns. Unlike traditional ML models, LLMs generate output token by token through an iterative process consisting of two distinct phases: prefill (generating the first token) and decode (generating subsequent tokens). This creates unique computational patterns where prefill operations are compute-heavy while decode operations are memory bandwidth-heavy, requiring specialized runtime optimizations. The team implemented continuous batching as a core optimization technique, using an analogy of a bus system where new passengers (requests) can board at each stop (decoding step) if there are empty seats. This approach maximizes GPU utilization by avoiding idle resources that would occur with static batching when shorter responses complete early. The system also implements KV (Key-Value) caching to avoid cubic attention computation complexity, storing intermediate attention states to enable quadratic scaling instead. For hardware utilization, Meta employs sophisticated parallelism strategies based on model size requirements. The 8B models fit comfortably on single GPUs, while 70B models require tensor parallelism across 2-8 GPUs to handle both weights and KV cache memory requirements. The massive 405B models necessitate pipeline parallelism across multiple nodes, with Meta recommending avoiding multi-node tensor parallelism due to communication overhead challenges. ## Performance Optimization and System Resource Management Meta's approach to performance optimization recognizes that LLM inference involves managing three critical system resources: compute capacity, memory bandwidth, and memory capacity. These resources scale differently with model size, sequence length, and batch size, creating optimization challenges since hardware resource ratios are fixed at manufacturing time. The team focuses on different latency metrics depending on use case requirements: Time to First Token (TTFT) for reducing user-perceived delays, Time to Token (TTIT) for streaming responsiveness, and end-to-end latency for batch processing scenarios. This nuanced approach to performance metrics enables targeted optimizations for specific product requirements. Disaggregation represents a significant architectural decision where Meta separates prefill and decode operations into different services. This allows independent scaling of compute-heavy prefill operations and memory-bandwidth-heavy decode operations, but introduces complexity in transferring hundreds of megabytes of KV cache data between services. The team had to implement sophisticated request scheduling and overlapped data transfer techniques to maintain acceptable performance. Context parallelism provides another dimension of scaling for extremely long inputs (128K tokens), further partitioning workloads at the context dimension to maintain responsiveness. However, this approach increases system complexity and failure blast radius, requiring dedicated job scheduling infrastructure. ## Advanced Optimization Techniques Meta implements hierarchical caching systems that leverage the memory hierarchy from HBM (hundreds of gigabytes) through DRAM (terabytes) to flash storage (tens of terabytes). Common system prompts are cached in high-speed HBM, active user chat histories in DRAM, and less frequently accessed data in flash storage. This approach commonly achieves over 50% reduction in both latency and capacity requirements, though it requires careful management of positional encoding challenges where identical tokens have different embeddings based on position. The team employs various model optimization techniques including prompt compression, smaller specialized models through fine-tuning and distillation, and quantization approaches. However, Meta takes a cautious approach to quantization, emphasizing that benchmark performance doesn't always translate to real-world product performance, requiring careful evaluation and gradual rollout processes. Speculative decoding, chunked prefill, attention kernel optimizations, and token-level sparsity represent additional optimization vectors, each with specific tradeoffs between different performance metrics and quality considerations. The team emphasizes that the effectiveness of these techniques depends heavily on specific workload characteristics. ## Production Challenges and Infrastructure Complexity Moving from prototype to production introduces significant complexity as request patterns become unpredictable with varying user engagement levels, input/output ratios, and temporal patterns including daily peaks and random evaluation spikes. Meta's production environment serves multiple types of workloads: real-time chatbots, human-in-the-loop annotation systems aligned with evaluator schedules, and batch processing for summarization and feature generation that can tolerate time-shifting. The team observes that mature products typically require serving multiple LLMs including main chat models and ancillary models for safety, planning, rewards, and function calling. This creates complex deployment orchestration requirements where peak FLOPS specifications from vendors become largely irrelevant, with systems commonly losing 50% effective FLOPS at the kernel level and up to 10x when considering latency bounds and operational buffers. Meta's end-to-end latency analysis reveals that LLM inference often represents a small portion of total request latency. Network roundtrips add 75ms, naive load balancing another 75ms, multimodal image downloads 150ms, and business logic coordination including safety checks and external API calls can add 400ms or more. This perspective shifts optimization focus from pure inference speed to holistic system performance. ## Scaling and Infrastructure Management At scale, Meta developed sophisticated deployment allocation systems that consider network topology, maintenance events, and hardware heterogeneity. The team created dedicated job schedulers for distributed inference and deployment solvers that treat autoscaling as a shard placement problem, understanding supply and demand dynamics even when demand exceeds available supply. Cost optimization requires extensive performance benchmarking automation and data science to create "inference manuals" that help teams choose optimal configurations based on latency requirements. Meta's analysis shows that numerous tail deployments collectively consume more GPU resources than primary models, making observability and automation critical for identifying optimization opportunities. The infrastructure supports traffic splitting and A/B testing across multiple model versions, adding another complexity dimension to all previously discussed challenges. This requires sophisticated model management, experimentation frameworks, and deployment orchestration capabilities. ## Organizational and Technical Integration Challenges Meta emphasizes that successful LLM serving requires deep integration across modeling, systems, and product teams. The challenge isn't simply hiring GPU experts or distributed systems specialists, but ensuring these diverse expertise areas communicate effectively and understand tradeoffs in common language. Approximately 70-80% of effort involves traditional systems engineering work building scalable, debuggable foundations rather than LLM-specific optimizations. The team works closely with PyTorch and hardware vendors, gathering signals from production deployments to inform kernel-level performance improvements and feature development. This collaboration extends to introducing new hardware like AMD GPUs to Meta's fleet, requiring coordination across multiple engineering teams. ## Results and Lessons Learned Meta's approach demonstrates that getting basics right (proper model runners, hardware matching, parallelism strategies) provides 10x foundation improvements. Distributed inference, model optimization, and caching techniques can provide additional 2-4x performance gains. Advanced techniques offer potential for further 10-100x improvements but require careful evaluation against specific workload requirements. The case study reveals that LLM serving infrastructure represents an iceberg where model and hardware considerations are merely the visible tip. The underlying infrastructure includes execution engines, production service capabilities (monitoring, routing, scheduling), and platform capabilities (allocation, deployment management, experimentation) that require substantial engineering investment. Meta's experience underscores the importance of continuous evaluation and testing, as inference bugs can manifest as subtle performance degradation rather than obvious failures. The team implements CI/CD practices with benchmark runs on every code change and comprehensive testing on releases. The presentation concludes by acknowledging that many scaling challenges remain unsolved, expressing hope for continued innovation in development and deployment tooling for GenAI applications. Meta's journey illustrates the substantial engineering complexity required to productionize LLMs at scale, requiring sophisticated software, algorithms, and cross-functional collaboration to achieve acceptable performance and cost efficiency.