Netflix built a comprehensive in-house LLM serving platform to run the full inference stack internally rather than relying on hosted APIs, addressing needs for low latency, deep customization, and integration with existing production infrastructure. The platform unified LLM serving with their existing Model Scoring Service (MSS), selecting vLLM as the inference engine and integrating it with NVIDIA Triton Inference Server, while exposing both gRPC and OpenAI-compatible HTTP APIs. Key challenges included engine version compatibility, constrained decoding at scale, and deployment strategies for GPU-based models. The solution enabled seamless experimentation-to-production workflows, handled diverse workloads including embeddings, prefill-only inference, autoregressive decoding, and custom constraint logic, with particular success in migrating from vLLM V0 to V1 to achieve batch-level constrained decoding that kept latency flat as batch sizes grew.
Netflix’s case study presents a comprehensive production LLM serving infrastructure built entirely in-house rather than relying on third-party hosted APIs. This decision represents a significant commitment to owning the full ML stack and reflects Netflix’s scale requirements, customization needs, and integration constraints. The platform was published in early 2026 (described as “5 days ago” from the publication date) and represents the culmination of work spanning at least through Q4 2025, when they migrated to vLLM V1.
The core architectural decision was to integrate LLM serving into their existing Model Scoring Service (MSS) infrastructure rather than creating a separate ML silo. This unified approach means that LLMs are treated like any other model type—XGBoost, TensorFlow, PyTorch—all served through the same gRPC interface, using the same client libraries, health checking, and deployment pipelines. However, they also recognized the ecosystem standardization around OpenAI-compatible APIs and added that as an additional frontend, enabling seamless migration from hosted models to self-hosted fine-tuned versions.
The serving architecture follows a layered approach with clear separation of concerns. At the top sits a unified JVM-based serving system that handles routing, A/B testing, candidate generation, feature fetching, inference orchestration, post-processing, and logging. This frontend supports both real-time and cached batch processing paths. Small CPU models run in-process within this serving system to avoid remote call overhead, while larger GPU models delegate inference to the remote Model Scoring Service.
MSS serves as the shared inference backend, supporting multiple model types behind a unified interface with NVIDIA Triton Inference Server managing the lower-level concerns of model loading, batching, and GPU scheduling. Above Triton, a Java control plane handles deployment, versioning, health checking, autoscaling, and multi-region rollout. This separation allows model authors to package artifacts and configure deployments while the control plane handles provisioning GPU instances, configuring Triton, and orchestrating zero-downtime upgrades.
The system exposes two primary paths for callers to reach inference: a gRPC path through the unified serving system (the traditional path for existing ML workloads) and a direct HTTP path using OpenAI-compatible APIs for newer LLM-driven applications. This dual-interface approach demonstrates pragmatism—maintaining consistency with existing infrastructure while adopting ecosystem standards where appropriate.
One of the most significant technical decisions was the choice of inference engine. The platform originally ran on TensorRT-LLM, which offered strong performance and was already integrated with Triton. However, by summer 2025, the landscape had shifted considerably. Open-source engines had largely closed the performance gap with specialized stacks, and Netflix’s workload requirements had diversified to include embedding generation, prefill-only inference for ranking and retrieval, autoregressive decoding, and custom models with complex per-step constraint logic.
After re-benchmarking against this diverse workload mix, they selected vLLM based on operational fit rather than raw performance alone. The key factors were: the ability to load custom model architectures without multi-step compilation pipelines (enabling faster iteration on non-standard models), extensibility hooks for custom decoding logic (critical for their constrained-decoding requirements), improved debuggability compared to compiled engines in earlier TensorRT-LLM versions, and familiarity within the research community (reducing the research-to-production handoff cost).
This decision reflects a mature LLMOps perspective that recognizes operational considerations—debuggability, flexibility, developer velocity—often matter as much as peak performance metrics, especially when performance differences have narrowed.
The integration of vLLM into Triton revealed important tradeoffs around model packaging. Triton supports two packaging approaches, each with significant implications for maintainability and the coupling between model artifacts and frontend upgrades.
The Python backend requires authors to define explicit input/output tensor specifications at packaging time. These specs are frozen into the artifact and must match what the frontend’s request builder expects. This creates tight coupling—every frontend upgrade touching I/O specs requires coordinated changes to packaging code, otherwise requests fail at runtime.
The vLLM backend takes a different approach: the artifact is just a JSON config pointing to model weights and tokenizer. Triton’s vLLM backend reads this config and generates I/O tensor specs dynamically at deployment time. The model author never defines them explicitly, and models and frontend can evolve independently.
Netflix chose the vLLM backend as the “architecturally correct default,” but production revealed two important gotchas. First, version mismatches between Triton and vLLM can cause complete backend failures. For example, Triton 25.09 importing vllm.engine.metrics—a module removed in vLLM 0.11.2—would cause the backend to fail to load entirely. This forced them to pin compatible versions in the service image and prevent model authors from overriding vLLM versions at packaging time.
Second, the vLLM backend assumes a standard HuggingFace-compatible model and handles the full inference lifecycle. Models requiring custom preprocessing, postprocessing, or non-standard execution patterns—ensemble pipelines, custom tokenization—must use the Python backend as an escape hatch. This suggests that even with good abstractions, production ML systems need flexibility for edge cases.
A key design principle was that LLM models should not be “special snowflakes” requiring different interfaces from other model types. Yet they also recognized that the OpenAI-compatible API has become the de facto standard across the LLM ecosystem—inference engines, orchestration frameworks, evaluation tools, and client libraries all speak it.
Their solution was to expose the OpenAI-compatible API as an additional frontend alongside the existing gRPC interface. This dual-interface approach pays dividends in the experimentation-to-production workflow: graduating from a hosted model to a fine-tuned self-hosted version (for quality, latency, cost, or data privacy reasons) becomes nearly seamless with minimal code changes.
Behind the API, they reuse NVIDIA’s Triton OpenAI-compatible frontend implementation, which starts an embedded Triton server, wraps it in a TritonLLMEngine that converts request schemas into Triton inference requests, and serves responses through FastAPI. KServe HTTP/gRPC frontends run alongside, keeping the same Triton instance accessible to the Java control plane over gRPC.
However, adopting Triton’s frontend directly exposed a critical gap: the response_format parameter (for structured output) was accepted by the schema but silently dropped before reaching vLLM. Callers requesting JSON output would proceed without guided decoding constraints and could receive malformed JSON with no error surfaced. They addressed this by using git-subtree to fork and patch the frontend, translating response_format into vLLM’s guided decoding parameters at request time. This illustrates the reality that even when leveraging open-source components, production requirements often necessitate customization and the infrastructure to maintain those customizations.
GPU deployments introduce unique challenges compared to CPU services: longer startup times and the potential for I/O schema changes between model versions create coordination problems. Netflix’s platform offers two deployment strategies to handle different scenarios.
Red-Black deployment runs a new version alongside the current one, shifts traffic in phases after health checks pass (with the new version scaling up while the old scales down at the same rate), and supports atomic rollback if any step fails. This works well when the model interface is stable, but production revealed a coordination gap: when a new version requires I/O schema changes (like new tensor dimensions), upstream consumers can’t update their config until the new model is fully live. During the migration window, they inevitably send “old” requests to the “new” deployment, causing failures.
Versioned deployment solves this by maintaining independent deployments for every (modelId, modelVersion) pair. Multiple versions serve simultaneously, decoupling model deployment from consumer updates. Consumers wait for the new version to be fully ready before switching their config while the old version continues serving legacy traffic. The platform cleans up older deployments after inactivity but always preserves the latest. The tradeoff is a temporary increase in GPU cost during the transition overlap.
Their recommendation is pragmatic: embed variable configurations (like tensor shapes) directly into the inference model to make it version-agnostic, enabling the cheaper Red-Black path. Reserve Versioned deployment for the rare cases where breaking interface changes are unavoidable. This guidance reflects operational learning about the total cost of ownership.
Beyond the major architectural decisions, Netflix documents two operational details that proved important in production but weren’t anticipated during the design phase.
The boot sequence for bringing up a vLLM-on-Triton instance involves several coordinated steps. For model caching, downloading large LLMs directly from S3 or Hugging Face at startup inflates cold-start latency beyond what schedulers tolerate. Their solution is to materialize models on Amazon FSx at the time of model announcement, so warm starts hit a high-performance filesystem instead of object storage. They also configure whether Triton runs as an embedded server inside the OpenAI-compatible frontend process (when consumers need that API) or standalone, determined per-deployment at packaging time.
The observability story required custom work to unify metrics from multiple sources. vLLM writes metrics to PROMETHEUS_MULTIPROC_DIR as .db files, while Triton reports server-level metrics through its own Prometheus endpoint. Neither is aware of the other, and Triton’s built-in bridge surfaces only 9 of 40+ vLLM metrics—missing critical ones like token throughput, KV cache utilization, and prefix cache hit rates. They added a lightweight HTTP proxy that merges both into a single /metrics endpoint: fetching Triton metrics via HTTP, reading vLLM metrics from disk using Prometheus’s MultiProcessCollector, and returning combined output. This allowed existing dashboards and alerts to work without modification, but the need for this integration layer highlights the challenges of composing open-source components.
Perhaps the most technically interesting aspect of this case study is their work on constrained decoding at scale. Some Netflix production workloads rely heavily on fine-grained control over token generation. Rather than applying business logic after inference—paying for invalid generations then retrying or repairing—they push constraints inside the decode loop so the model generates compliant outputs by construction.
They implement this via vLLM’s custom logits processor interface, modeling each constraint as a state machine that evolves with generated token history and emits token-eligibility masks at each step. Each request gets its own configured processor since different requests apply different rules.
Their journey getting this to scale spanned two major vLLM versions. The initial implementation on vLLM V0 worked functionally but hit a scaling bottleneck. In V0, custom logits processors run per-request: the GPU produces logits for the whole batch, the CPU copies them across and waits for the transfer, then constraint logic runs sequentially for each request. This sequential execution happens because Python’s Global Interpreter Lock (GIL) prevents parallelizing the per-request work. CPU time in logit processing grows linearly with batch size, causing tail latencies to spike. The system becomes CPU-bound even though the model’s forward pass batches efficiently on GPU—a bottleneck invisible in single-request benchmarks that only surfaces under realistic concurrency.
The structural fix arrived in vLLM V1, which moved logits processing to batch level. They rewrote their custom processor to operate on batch-level data structures, computing masks across many requests together, and reimplemented the hot path in C++ with multi-threading to sidestep the GIL. The V1 API is more complex—requiring explicit tracking of batch membership changes via update_state(batch_update)—but necessary to maintain correct state in a dynamically evolving batch. Their performance data shows logits processing time staying flat as batch size grows, validating the architectural shift.
However, moving from performance bottlenecks to production readiness revealed two additional issues. First, vLLM V1 performs chunked prefilling, so a request can be prefilled over multiple engine steps. BatchUpdate lacks the granularity to indicate whether a request was fully or only partially prefilled, forcing them to add internal tracking. Second, under memory pressure, vLLM may evict a partially completed request’s KV cache and reschedule it later with a different prompt and output token list. This breaks the state machine’s assumption that output token lists grow monotonically. They detect when token history shrinks between decode steps, reset the state machine, and reinitialize from the new prompt.
This constrained decoding work exemplifies sophisticated LLMOps: identifying that business requirements (output constraints) should be enforced at the inference level rather than post-processing, implementing custom logic that integrates deeply with the inference engine, migrating that logic across major engine versions while preserving functionality, and hardening against operational edge cases like preemption and chunked prefilling that only surface in production under real workload patterns.
This case study demonstrates mature LLMOps engineering, but it’s important to note the context and tradeoffs. Netflix has exceptional scale, sophisticated existing ML infrastructure, and deep engineering resources. Their decision to run the full LLM stack in-house rather than using hosted APIs makes sense for their requirements—member-scale workloads, customization needs, data privacy considerations, and integration with existing systems—but would be overkill for many organizations.
The technical choices reflect this context: selecting vLLM over TensorRT-LLM prioritized operational concerns (debuggability, flexibility, iteration speed) over potentially higher peak performance, which is appropriate when performance is “good enough” and operational efficiency matters more. Their dual-interface approach (maintaining gRPC for existing systems while adding OpenAI-compatible APIs) shows pragmatic architecture that balances consistency with ecosystem adoption.
The challenges they encountered—version pinning requirements, silent API gaps, coordination issues during deployment, observability integration—are realistic problems that most organizations deploying LLMs at scale will face. Their solutions (pinning compatible versions, patching frontends, multiple deployment strategies, custom metrics aggregation) are practical but require maintenance overhead.
The constrained decoding work is technically impressive but also reflects Netflix’s specific workload requirements. Many organizations might find it sufficient to apply constraints via prompt engineering or post-processing rather than implementing custom logits processors in C++ with multi-threading. The investment makes sense for Netflix because they have production workloads where generating compliant outputs by construction provides significant business value.
Netflix indicates their next investments will focus on system prompt compression (reducing prompt length without sacrificing quality), asynchronous scheduling in vLLM V1, vectorized logits processors that run as fused GPU kernels instead of CPU code, and lower-precision model variants to decrease memory footprint and increase throughput. These priorities suggest continued optimization along multiple dimensions: cost (prompt compression, lower precision), performance (async scheduling, GPU kernels), and efficiency (memory optimization).
The commitment to working closely with the open-source community as the space evolves is noteworthy and suggests they view their infrastructure as benefiting from and contributing to open-source tooling rather than building entirely proprietary solutions.
This case study provides valuable insight into production LLM infrastructure at scale, demonstrating sophisticated engineering that goes well beyond simply deploying models. The emphasis on operational concerns—version management, observability, deployment strategies, debuggability—reflects the reality that running LLMs in production involves solving systems engineering problems as much as ML problems. The detailed discussion of tradeoffs and production-discovered issues provides honest insight rare in technical blog posts that often focus only on successes.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
This case study examines Cursor's implementation of reinforcement learning (RL) for training coding models and agents in production environments. The team discusses the unique challenges of applying RL to code generation compared to other domains like mathematics, including handling larger action spaces, multi-step tool calling processes, and developing reward signals that capture real-world usage patterns. They explore various technical approaches including test-based rewards, process reward models, and infrastructure optimizations for handling long context windows and high-throughput inference during RL training, while working toward more human-centric evaluation metrics beyond traditional test coverage.
Volkswagen Group Services partnered with AWS to build a production-scale generative AI platform for automotive marketing content generation and compliance evaluation. The problem was a slow, manual content supply chain that took weeks to months, created confidentiality risks with pre-production vehicles, and faced massive compliance bottlenecks across 10 brands and 200+ countries. The solution involved fine-tuning diffusion models on proprietary vehicle imagery (including digital twins from CAD), automated prompt enhancement using LLMs, and multi-stage image evaluation using vision-language models for both component-level accuracy and brand guideline compliance. Results included massive time savings (weeks to minutes), automated compliance checks across legal and brand requirements, and a reusable shared platform supporting multiple use cases across the organization.
