Baseten, an inference infrastructure provider, documented their engineering efforts to build and optimize production APIs for the GLM-5.2 model, achieving state-of-the-art performance with speeds up to 280 tokens per second. The company addressed the challenge of serving large language models with optimal latency and throughput by implementing multiple optimization strategies including scheduler improvements, speculative decoding, custom parallelism configurations, and batch size tuning. Their work resulted in benchmark-leading performance on both Time to First Token (TTFT) and Tokens Per Second (TPS) metrics, with their fast API variant specifically optimized for latency-sensitive use cases like coding and agents, demonstrating more than double the performance compared to initial launch-day implementations.
This case study from Baseten provides valuable insights into the production engineering challenges of serving large language models at scale, specifically focusing on the GLM-5.2 model released in June 2026. Baseten positions itself as an inference infrastructure provider that offers both general-purpose and optimized APIs for frontier models. The document, published July 26, 2026, describes a month-long optimization journey following GLM-5.2’s release, during which they achieved benchmark-leading performance metrics.
The case study is notable for its transparency about the engineering tradeoffs involved in LLM serving infrastructure, though readers should be aware this is marketing content from an infrastructure vendor and some claims about “fastest” performance should be viewed with appropriate skepticism. The company claims their GLM-5.2 API achieved peak speeds of 280 tokens per second with average speeds around 100 TPS, with performance more than doubling from launch day to the time of writing.
The fundamental LLMOps challenge addressed here is the multi-objective optimization problem inherent in serving large language models in production. Production LLM systems must balance several competing concerns: latency (how quickly responses begin and complete), throughput (how many requests can be processed), cost efficiency (hardware utilization and pricing), and quality (maintaining model output fidelity). Baseten’s solution demonstrates that different use cases require different positions along the Pareto frontier of these tradeoffs.
The company identified specific market demand for latency-optimized serving, particularly for use cases like coding assistants and agent-based applications where response time is critical. This led them to develop two distinct API offerings for the same model: a general-purpose API balanced for throughput and a “fast” API specifically tuned for minimal latency.
The infrastructure foundation for both APIs consists of NVIDIA B200 GPUs, representing cutting-edge hardware for LLM inference. This is significant from an LLMOps perspective as it demonstrates the importance of infrastructure decisions in production deployments. The choice of B200 GPUs suggests Baseten is investing in the latest hardware to achieve competitive performance, though this also raises questions about accessibility and cost for organizations without access to such advanced hardware.
The case study mentions their inference engine and “stack” more broadly, suggesting a comprehensive software layer sitting atop the hardware. While specific details about the inference engine are limited, the text references multiple components including a scheduler, weight management systems, and the overall serving architecture.
Baseten’s optimization work proceeded along two parallel tracks: improvements benefiting both APIs and specific optimizations for the fast API variant. The general improvements that enhanced both offerings included several key areas.
Scheduler optimization represents a foundational improvement to their serving infrastructure. While the document doesn’t detail the specific scheduler algorithms, scheduler optimization in LLM serving typically involves intelligent request batching, priority management, and resource allocation. The mention of “slightly increasing throughput” suggests these were incremental but meaningful improvements that likely involved better request packing and GPU utilization.
Weight quantization improvements came through the deployment of NVFP4 weights. FP4 (4-bit floating point) represents an aggressive quantization strategy that reduces memory footprint and can accelerate inference. The term “improved” NVFP4 weights suggests they refined their quantization approach, possibly through better calibration or quantization-aware techniques. This is a critical LLMOps consideration as quantization can significantly impact both performance and quality, requiring careful validation in production.
Speculative decoding enhancements represent one of the more sophisticated optimization techniques mentioned. Speculative decoding is a technique where a smaller, faster “draft” model generates candidate tokens that are then verified by the larger target model. When the draft model’s predictions are correct, multiple tokens can be generated in parallel, significantly reducing latency. The case study mentions both an initial speculative decoding profile and plans for further improvements, indicating this is an ongoing area of optimization. The effectiveness of speculative decoding depends heavily on the quality of the draft model and the acceptance rate of speculated tokens, making it a complex production consideration.
Bug fixes across both quality and performance represent the unglamorous but essential work of production LLM operations. Quality bugs in inference systems can manifest as incorrect outputs, while performance bugs might cause unexpected latency spikes or throughput degradation. The explicit mention of bugs “throughout the stack” suggests the complexity of production LLM systems and the need for comprehensive testing and monitoring.
The fast API variant represents a deliberate repositioning along the latency-throughput tradeoff curve. The two major configuration changes illustrate fundamental principles of distributed LLM serving.
Parallelism strategy adjustments constitute perhaps the most significant architectural difference. The general API uses Attention Data Parallelism (ADP) to improve throughput, while the fast API uses only Tensor Parallelism and Expert Parallelism configured specifically for latency. This deserves deeper analysis from an LLMOps perspective.
Attention Data Parallelism is a technique where different attention heads or data samples can be processed in parallel across GPUs, generally favoring throughput by allowing more requests to be processed simultaneously. Tensor Parallelism, by contrast, splits individual tensors across multiple GPUs, requiring tight synchronization but reducing per-request latency. Expert Parallelism is relevant for Mixture of Experts (MoE) architectures, distributing different expert networks across devices. The choice to use only Tensor and Expert Parallelism for the fast API suggests they’re prioritizing minimal per-request latency even at the cost of overall system throughput. This is a sophisticated production decision that requires deep understanding of both the model architecture and the serving workload.
Batch size reduction represents the second major configuration change. Batching is fundamental to efficient GPU utilization—processing multiple requests together amortizes fixed costs and improves throughput. However, larger batches increase queuing delays and can cause head-of-line blocking where fast requests wait for slow ones. The “substantial reduction in max batch size” for the fast API means fewer requests compete for GPU resources simultaneously, reducing latency but also reducing overall throughput. This explicitly trades efficiency for responsiveness.
Critically, the fast API charges 50% higher prices for both input and output tokens despite running on the same hardware. This pricing decision reflects the economic reality of the throughput-latency tradeoff: by serving fewer requests per GPU simultaneously, Baseten incurs higher per-request infrastructure costs. From an LLMOps perspective, this demonstrates the importance of workload-appropriate infrastructure choices and the need for differentiated service tiers in production LLM platforms.
The pricing strategy also reveals market dynamics: Baseten explicitly states they received “strong signal from the market that there is willingness to pay for more performance.” This suggests successful LLMOps involves not just technical optimization but also product positioning and customer segmentation based on performance requirements.
The case study heavily emphasizes benchmark results from Artificial Analysis, a third-party benchmarking service. The specific benchmarks measured on July 25, 2026 at approximately 7:00 PM Pacific Time show Baseten achieving leading performance on both Time to First Token (TTFT) and Tokens Per Second (TPS), as well as end-to-end response time.
Several important LLMOps considerations emerge around benchmarking. First, Baseten acknowledges that “LLM performance varies substantially based on the amount of traffic in a system, the pattern of said traffic, and the input and output sequence lengths.” This is a crucial production reality often overlooked in benchmark-focused discussions. The Artificial Analysis benchmark uses approximately 10,000 input tokens generating approximately 1,000 output tokens, which may not represent all production workloads.
Second, the emphasis on both benchmark performance and “real-world usage” suggests Baseten understands the limitations of synthetic benchmarks. They mention receiving “positive feedback from the market around our API’s leading performance in real-world usage, not just benchmarks,” though specific metrics or case studies aren’t provided. In production LLMOps, real-world performance monitoring and customer-facing SLAs are ultimately more important than benchmark leaderboards.
The case study concludes by noting ongoing optimization work, including another planned improvement to speculative decoding and learnings from building APIs for other models like Kimi K3 that will be applied back to GLM-5.2. This reveals an important aspect of production LLMOps: optimization is never finished. As models evolve, hardware advances, and new techniques emerge, production infrastructure requires continuous refinement.
The mention of cross-model learning (applying insights from Kimi K3 to GLM-5.2) suggests Baseten is building generalized optimization expertise rather than one-off solutions. This scalable approach to production optimization is essential for infrastructure providers serving multiple models.
While this case study provides valuable technical insights, several caveats warrant mention. As marketing content from an infrastructure vendor, claims about having “the fastest API in the world” or achieving “SOTA speeds” should be interpreted carefully. Performance is workload-dependent, and what’s fastest for one use case may not be optimal for another. The benchmark results shown are snapshots in time; competitive dynamics in LLM serving are rapidly evolving.
The document lacks certain details that would be valuable for comprehensive LLMOps understanding, including specific information about monitoring and observability systems, error handling and fallback strategies, model quality validation processes, and cost-efficiency metrics beyond simple per-token pricing. Real production systems require sophisticated monitoring, alerting, and incident response capabilities that aren’t discussed here.
Additionally, the focus on a single model (GLM-5.2) raises questions about generalizability. Some optimizations described may be model-specific, and the techniques might not transfer equally well to different architectures or model families.
Despite these limitations, the case study illustrates several important production principles. The explicit recognition of latency-throughput tradeoffs and the creation of differentiated service tiers demonstrates sophisticated production thinking. Not all workloads have the same requirements, and one-size-fits-all infrastructure is suboptimal.
The multi-layered optimization approach—spanning hardware selection, parallelism strategies, batching configurations, quantization, and algorithmic improvements like speculative decoding—shows that production LLM performance requires optimization across the entire stack. No single technique dominates; meaningful improvements come from careful attention to multiple components.
The emphasis on both benchmark performance and real-world usage highlights the importance of comprehensive testing and validation. Synthetic benchmarks provide useful comparison points but don’t capture all production behaviors.
Finally, the continuous improvement mindset and cross-model learning approach reflect the reality that LLMOps is an ongoing discipline, not a one-time implementation. As the field rapidly evolves, production systems must evolve with it through sustained engineering investment.
Shopify's CTO discusses how the company has achieved near-universal AI adoption internally, with nearly 100% of employees using AI tools daily as of December 2025. The company has developed sophisticated internal platforms including Tangle (an ML experimentation framework), Tangent (an auto-research loop for automatic optimization), and SimGym (a customer simulation platform using historical data). These systems have enabled dramatic productivity improvements including 30% month-over-month PR merge growth, significant code quality improvements through critique loops, and the ability to run hundreds of automated experiments. The company provides unlimited token budgets to employees and emphasizes quality token usage over quantity, focusing on efficient agent architectures with critique loops rather than many parallel agents. They've also implemented Liquid AI models for low-latency applications, achieving 30-millisecond response times for search queries.
Baseten has built a production-grade LLM inference platform focusing on three key pillars: model-level performance optimization, horizontal scaling across regions and clouds, and enabling complex multi-model workflows. The platform supports various frameworks including SGLang and TensorRT-LLM, and has been successfully deployed by foundation model companies and enterprises requiring strict latency, compliance, and reliability requirements. A key differentiator is their ability to handle mission-critical inference workloads with sub-400ms latency for complex use cases like AI phone calls.
Poolside AI, a foundation model company focused on code generation, developed a comprehensive "Model Factory" system that enables them to train and deploy models from scratch to production in 5-8 weeks with a team of fewer than 70 researchers. Their approach treats model building as 90% engineering, emphasizing automation, reproducibility, and rapid experimentation (10,000-20,000 experiments per month). The result is the Laguna S model (118B parameters, 8B active), which demonstrates that smaller models with better behaviors—persistence, verification, and backtracking—can compete with models 10x their size, suggesting a path toward commoditized, open-weight foundation models.
