## Overview This case study from OpenAI describes the infrastructure challenges and solutions developed to scale access to two of their LLM-based products: Codex (a code generation system) and Sora (a video generation system). The core problem emerged from rapid adoption that exceeded initial capacity expectations, creating a tension between providing fair access, maintaining system performance, and ensuring positive user experiences. Rather than simply imposing hard rate limits or moving to pure usage-based billing, OpenAI developed a hybrid real-time access control system that intelligently blends rate limits with a credit-based continuation mechanism. The case study is particularly valuable from an LLMOps perspective because it addresses a critical production challenge that many organizations deploying LLMs at scale will encounter: how to manage access and resource consumption in a way that balances fairness, system stability, user experience, and business viability. It's worth noting that this is OpenAI's perspective on their own solution, and while the technical details appear sound, organizations should consider whether this level of complexity is necessary for their own use cases or whether simpler approaches might suffice. ## The Core Problem: Limitations of Traditional Access Models OpenAI identified that traditional access control mechanisms created a false dichotomy that didn't serve their needs well. Rate limits, while useful for demand smoothing and ensuring fair access, created frustrating "come back later" experiences when users were actively engaged and deriving value. On the other hand, pure usage-based billing from the first token discouraged exploration and experimentation, which was particularly problematic for products like Codex where users need to try things out to understand value. The challenge was compounded by the interactive nature of both products. Codex is used in development workflows where interruptions break concentration and momentum. Sora generates video content where creative work benefits from continuous iteration. In both cases, hitting a hard stop at an arbitrary limit disrupts the value creation process. Simply raising rate limits wasn't viable because it would eliminate important capacity controls and fairness mechanisms, potentially allowing some users to monopolize resources at the expense of others. From an LLMOps perspective, this highlights a unique challenge with LLM-based products compared to traditional API services: the highly variable and potentially expensive nature of individual requests combined with the interactive, creative workflows that benefit from continuous access. A single Sora video generation or complex Codex query can consume significantly more resources than simpler requests, making capacity planning and access control particularly challenging. ## The Conceptual Solution: Access as a Decision Waterfall OpenAI's key conceptual innovation was reframing access control from a binary gate ("allowed or not") to a decision waterfall that asks "how much is allowed, and from where?" This creates a hierarchy of access mechanisms that the system evaluates in sequence for each request: The system checks rate limits first, allowing requests to proceed if limits haven't been exceeded. When rate limits are exhausted, it seamlessly transitions to checking credit availability and consuming credits if available. From the user's perspective, there's no visible transition between these modes—they simply continue working while the system handles the complexity of determining which access mechanism applies. This waterfall model also accommodates additional layers like free tiers, promotional credits, and enterprise entitlements, all within a unified decision framework. The elegance of this approach is that it matches how users actually experience the product: they don't think about "switching systems" or "changing modes," they simply use Codex or Sora until they either complete their work or run out of all available access mechanisms. However, it's important to note that this conceptual model comes with significant implementation complexity. Organizations considering similar approaches should carefully weigh whether their scale and user base justify the engineering investment required, or whether simpler tiered plans or pure usage billing might be more appropriate. ## Architecture: Real-Time Decision Making with Eventual Balance Updates The technical architecture reflects a careful balance between different requirements. OpenAI needed real-time decision making to avoid frustrating user experiences, but also required provably correct billing to maintain trust, particularly with enterprise customers. They resolved this tension through a split architecture: synchronous access decisions combined with asynchronous balance updates. When a request arrives, the system makes a real-time synchronous decision by checking rate limit windows and verifying sufficient credit availability. This ensures users get immediate feedback about whether their request will proceed. However, the actual debit of credit balances happens asynchronously through a streaming processor. This design choice prioritizes user experience (no waiting for balance updates) and correctness (complete audit trails) over strict real-time enforcement of credit balances. The tradeoff is that the system can briefly allow users to exceed their credit balance between when a request is approved and when the balance update is processed. OpenAI explicitly accepts this and automatically refunds any overshoots, choosing to err on the side of user satisfaction rather than strict enforcement. This is a pragmatic decision for their use case, though organizations with different requirements (such as strict budget controls) might make different choices. The distributed nature of the system handles per-user, per-feature usage tracking, maintains sliding windows for rate limits, tracks real-time credit balances, and processes debits idempotently. Every request passes through a single evaluation path that produces one definitive outcome, ensuring consistency across products and eliminating the risk of different services making conflicting decisions. ## Provably Correct Billing: The Foundation of Trust A particularly sophisticated aspect of OpenAI's implementation is the emphasis on provable correctness in billing. This reflects both technical rigor and business reality—enterprise customers and regular users alike need assurance that they're being charged accurately. The system maintains three separate but connected datasets that form a complete audit trail: Product usage events capture what users actually did, regardless of whether it resulted in charges. These events provide the foundational truth about user activity and enable OpenAI to explain why credits were or weren't consumed for any given request. Every event carries a stable idempotency key, which is crucial for preventing double-charging in the face of retries, replays, or system failures—a common challenge in distributed systems. Monetization events represent charges associated with usage, translating actual product interactions into billing impacts. These are derived from product usage events but represent a separate concern: the business logic of what should be charged. Balance updates track how user credit balances change over time and why. Critically, these updates are performed atomically with the balance modification itself—the system decreases the credit balance and inserts a balance update record in a single database transaction. This guarantees that every change to a user's balance has a corresponding audit record that can be traced back through monetization events to the original product usage. The separation of these datasets enables independent auditing, replay, and reconciliation of every layer. OpenAI can verify that their system is functioning correctly by comparing these datasets offline, catching any inconsistencies that might indicate bugs or implementation issues. This is an intentional architectural decision where they prioritize provable correctness over immediate balance updates, accepting a small delay in balance visibility in exchange for trustworthy billing. Balance updates are serialized per account, preventing race conditions where concurrent requests might try to spend the same credits. This is a critical correctness property in a distributed system where multiple requests for the same user might arrive simultaneously. ## Why In-House Rather Than Third-Party Solutions OpenAI evaluated existing usage billing and metering platforms but found them insufficient for two critical requirements: real-time decision making and integrated transparency. Third-party billing platforms are typically designed for invoicing and reporting—they excel at accurately tracking usage over time and generating bills, but they don't support the immediate "can this request proceed right now?" decisions that interactive products require. When a user hits a rate limit and has credits available, OpenAI's system must know immediately whether to allow the request to proceed. Best-effort or delayed counting would manifest as surprise blocks (users with available credits being denied access), inconsistent balance displays, and incorrect charges—all of which damage user trust and experience in interactive products like Codex and Sora. The transparency requirement was equally important. Users need to understand why a request was allowed or blocked, how much usage it consumed, and which limits or balances were applied. This requires tight integration with the decision waterfall rather than being solved in isolation. A separate billing platform would only see one piece of the puzzle (credit consumption) without context about rate limits, free tiers, or other access mechanisms. From an LLMOps perspective, this decision reflects a broader principle: when user experience and system behavior are tightly coupled to operational infrastructure, and when the scale is sufficient, building in-house may be necessary. However, organizations should carefully consider whether they truly need this level of control and integration. For many use cases, simpler approaches or existing platforms might be entirely adequate, and the engineering investment required to build and maintain such a system is substantial. ## Production Operational Considerations While the case study focuses primarily on architectural design, it implicitly reveals several LLMOps operational considerations. The system must handle high request volumes with low latency, as synchronous decision-making requires fast lookups of rate limit windows and credit balances. This likely requires careful database selection, caching strategies, and optimization of hot paths. The idempotency mechanism is essential for operational reliability. In any distributed system, requests may be retried due to network issues, timeouts, or other failures. Without stable idempotency keys and careful deduplication logic, retries could result in double-charging users, which would quickly erode trust. This is a fundamental LLMOps concern: production LLM systems must be resilient to the various failures that occur in distributed environments while maintaining correct behavior from the user's perspective. The asynchronous balance update mechanism requires reliable stream processing infrastructure. The streaming processor that consumes usage events and applies balance updates must be highly available and must not lose events, as this would result in charges not being applied and balances becoming incorrect. The system must handle back-pressure, ensure ordered processing per user, and gracefully handle failures and restarts without creating duplicate charges. Monitoring and observability are implicitly critical, though not deeply discussed in the case study. OpenAI mentions the ability to "independently audit, replay, and reconcile every layer," which suggests comprehensive logging and instrumentation. In production, they likely monitor metrics like decision latency, balance update lag, rate limit hit rates, credit consumption rates, and discrepancies between datasets. These metrics would be essential for understanding system health and user behavior. ## Balancing User Experience with System Constraints A key LLMOps insight from this case study is how OpenAI balances user experience with system constraints. Rather than simply imposing technical limitations on users (hard rate limits), they created a system that accommodates continued usage while still managing capacity and fairness. This reflects a user-centric approach to operations where the infrastructure adapts to user needs rather than forcing users to adapt to infrastructure limitations. The choice to allow brief credit balance overshoots rather than strictly enforce limits in real-time is particularly interesting. OpenAI explicitly chooses to prioritize user momentum over perfect enforcement, automatically refunding overshoots rather than blocking requests. This is a sophisticated operational decision that recognizes the asymmetric costs: blocking a user with available credits creates immediate frustration and disrupts creative work, while briefly allowing an overshoot that's quickly refunded has minimal business impact. However, organizations should carefully consider whether this approach fits their context. In environments with strict budget controls or cost sensitivity, preventing overshoots might be more important than maximizing user momentum. The right balance depends on the specific business model, user expectations, and cost structure. ## Extensibility and Future Direction OpenAI notes that this foundation can extend to more products over time, with Codex and Sora being just the beginning. This suggests they've built a general-purpose access control and usage tracking platform rather than a product-specific solution. From an LLMOps perspective, this represents important infrastructure that can support an entire ecosystem of LLM-based products. The waterfall model naturally accommodates additional layers and mechanisms. Organizations could add trial periods, educational discounts, enterprise volume commitments, or temporary capacity boosts during low-demand periods—all within the same decision framework. This extensibility is valuable as product offerings evolve and business models mature. However, building such a general-purpose system requires significant upfront investment and ongoing maintenance. Organizations should consider whether they need this flexibility or whether simpler, product-specific solutions might be more appropriate initially, with consolidation and generalization happening later as patterns emerge. ## Critical Assessment and Considerations While OpenAI's solution is technically sophisticated, it's important to view this case study with appropriate context. This is a first-party account of their own infrastructure, written to showcase their engineering capabilities. The actual implementation likely encountered challenges, edge cases, and operational issues that aren't discussed in a marketing-oriented blog post. The level of complexity described here is probably only justified at OpenAI's scale and with their specific product requirements. Many organizations deploying LLMs in production would be better served by simpler approaches: straightforward rate limiting, tiered subscription plans, or standard usage-based billing. The engineering investment required to build, test, and operate a system with real-time access decisions, distributed usage tracking, provably correct billing, and comprehensive audit trails is substantial. The case study also doesn't discuss failure modes, operational challenges, or lessons learned from incidents. In production, complex distributed systems inevitably encounter race conditions, consistency issues, performance degradation, and unexpected behaviors. Understanding how OpenAI handles these situations would be valuable but isn't covered. Additionally, while the emphasis on provable correctness and audit trails is admirable, it's worth questioning whether the asynchronous balance update design truly achieves the stated goals. The brief window where balances can be overspent represents a potential consistency issue that's resolved through refunds rather than prevention. For most use cases this is fine, but organizations with different requirements might need stricter consistency guarantees. ## Key Takeaways for LLMOps Practitioners This case study illustrates several important principles for operating LLMs in production at scale. First, access control and rate limiting for LLM products require more sophistication than traditional API services due to the interactive nature of use cases, the high variability in request costs, and the importance of user momentum in creative workflows. Second, user trust is paramount when charging for LLM usage. The investment in provably correct billing, comprehensive audit trails, and transparency reflects the reality that users need confidence in the fairness and accuracy of consumption tracking. This is especially true for expensive operations like code generation or video creation where costs can accumulate quickly. Third, architectural decisions should optimize for user experience within system constraints rather than simply imposing technical limitations. OpenAI's choice to seamlessly blend rate limits and credits, allow brief overshoots, and make real-time decisions all prioritize keeping users in their flow state rather than forcing them to think about infrastructure limitations. Finally, the decision to build in-house versus using third-party solutions should be based on specific requirements around latency, integration, and control. For OpenAI's use case, the need for real-time decisions tightly integrated with their product experience justified the investment. Other organizations should carefully assess whether their requirements truly necessitate similar complexity or whether simpler approaches would suffice.