## Overview Roblox operates one of the world's largest user-generated content platforms, serving an average of 97.8 million daily active users who collectively generate approximately 6.1 billion chat messages and 1.1 million hours of voice communication per day across 28 languages. This case study details how Roblox has built and scaled AI-driven content moderation systems over approximately five years to handle this massive volume of content in real-time. The company's approach represents a significant LLMOps challenge: deploying and continuously improving multiple ML models that must operate at extreme scale (over 750,000 requests per second), with millisecond latency requirements, across multiple modalities (text, voice, images, 3D assets), while maintaining high accuracy and adapting to rapidly evolving language patterns, slang, and adversarial evasion techniques. The case study emphasizes that while the company presents their system as highly successful, the necessity for such extensive infrastructure reveals the inherent challenges of content moderation at scale. From February to December 2024, users uploaded approximately 1 trillion pieces of content, with approximately 0.01% violating policies—though this small percentage still represents millions of violations that needed detection and removal. The system's core principle is that AI is only deployed when it performs significantly higher in both precision and recall than humans at scale, with human moderators retained for nuanced cases, appeals, and continuous system improvement. ## Technical Architecture and Scale Challenges Roblox's moderation infrastructure centers on large transformer-based models with knowledge across various modalities, which are then optimized for production deployment through distillation and quantization techniques. The evolution of their text filtering system illustrates the progression of their LLMOps maturity: beginning with rule-based filters over a decade ago, advancing to transformer-based filters approximately five years ago, and now operating multiple purpose-built models for different violation types running on specialized GPU-based infrastructure. A specific example demonstrates the extreme scale requirements: their personally identifiable information (PII) filter was processing so many requests per second on their existing CPU-based serving stack that it became unsustainable. The team built an entirely new GPU-based serving stack leveraging their "cellular infrastructure" (though the case study doesn't elaborate on what this architecture entails). The optimization process involved separating tokenization from inference and then accelerating inference through quantization and distillation of larger models. These improvements quadrupled their requests per second capacity, with the PII filter now handling 370,000 RPS at peak. Critically, this optimization didn't just improve throughput—it also reduced false positives by 30% and increased PII detection by 25% across all supported languages, demonstrating that model optimization and performance improvements can go hand-in-hand rather than being in conflict. The text filtering models collectively handle more than 750,000 RPS, processing the 6.1 billion daily chat messages. This requires not just raw computational power but sophisticated orchestration of multiple specialized models, each optimized for different violation types such as PII, profanity, hate speech, and other policy violations. The case study suggests these are purpose-built models rather than a single general-purpose classifier, indicating a deliberate architectural decision to separate concerns and optimize each model for its specific detection task. ## Voice Moderation and Real-Time Feedback Systems Voice moderation presents different technical challenges than text, as voice communications cannot be blocked pre-emptively in the same way text can be filtered before display. Roblox's voice safety classifier processes voice communications within 15 seconds across eight languages, operating at up to 8,300 RPS at peak. The company has open-sourced this model as part of their commitment to sharing safety innovations, which provides some external validation of their approach though the case study doesn't detail what independent assessments have found. The latest version of their voice classifier reportedly achieves a recall rate that's 92% higher than their initial version with a 1% false positive rate. However, the case study doesn't specify the absolute recall rate or provide context about the base rate of violations in voice communications, making it difficult to fully assess these claims. A 92% improvement in recall could mean going from 50% to 96% recall, or from 5% to 9.6% recall—the practical implications differ dramatically. Rather than blocking voice communications, Roblox implements real-time feedback through on-screen notifications that educate users about policy violations and offer opportunities to appeal. This design choice reflects a balance between safety and user experience, acknowledging that real-time voice communication requires a different intervention strategy than text filtering. The company reports that warning notifications have "effectively changed user behavior and increased both civility and engagement," though specific metrics beyond the stated percentages aren't provided. Their research indicates that suspensions impact behavior for up to three weeks afterward, reducing reoffense rates and user reports. Building on voice chat success, Roblox has implemented real-time feedback for text chat as well. Recent experiments showed that in-experience text chat notifications and time-outs resulted in a 5% reduction in filtered chat messages and a 6% reduction in consequences from abuse reports. These relatively modest improvements suggest that while real-time feedback has positive effects, it's not a silver bullet—the majority of user behavior remains unchanged by these interventions alone. ## Data Quality and Continuous Improvement Pipeline A significant portion of the case study focuses on data quality and the continuous improvement pipeline, which represents critical LLMOps practices for maintaining model performance over time. Roblox emphasizes that their systems are trained to optimize for fewer false negatives (erring on the side of removing potentially violating content) while continuously working to minimize false positives (which frustrate users when compliant content is removed). This represents a fundamental tension in content moderation: safety versus user experience. The company employs multiple strategies for building robust training and evaluation datasets: **Human-curated golden sets**: Policy experts hand-curate examples that most closely match issues the system should detect. These serve as high-quality ground truth data for both training and evaluation. **Uncertainty sampling**: The system samples edge cases where models were previously confused, focusing improvement efforts on the decision boundary where the model is least confident. This is a well-established active learning technique that efficiently identifies the most informative examples for labeling. **AI-assisted red teams (AARTs)**: These simulate adversarial attacks to probe for weaknesses in the moderation system, generating examples of evasion techniques that adversarial users might attempt. This proactive approach helps the system stay ahead of users who deliberately try to circumvent filters. **Appeals process**: When users request additional review and decisions are overturned, those examples become part of the dataset to improve future decisions. This creates a feedback loop where false positives directly contribute to model improvement. **User reporting system**: With tens of millions of potential reporters, Roblox has improved their reporting tool to allow users to capture entire scenes with avatar and object IDs and highlight specific concerns. Approximately 15% of eligible reports now provide visual annotations, and this additional context helps proactively identify problematic experiences. The company is exploring automatically creating AI-driven rules from user reports to increase responsiveness between model training cycles. **Synthetic data generation**: Large language models generate artificial data examples and labels that emulate real-world examples. This is particularly valuable for rare or edge cases where insufficient real examples exist. The ability to generate millions of synthetic examples addresses the long-tail problem in content moderation, though the case study doesn't discuss validation strategies to ensure synthetic data accurately represents real-world distributions or potential biases introduced by the generating LLMs. The evaluation process emphasizes two key metrics: alignment and quality. Alignment testing sends the same examples to multiple human moderators to check for consistency—if agreement is 80% or greater, moderators can make consistent decisions at scale. Lower alignment indicates policy or training confusion requiring iteration. Quality testing sends the golden set to moderators to verify correct decisions, with expert sampling to validate enforcement accuracy. High alignment and high quality indicate that policy can be enforced correctly and consistently; otherwise, both policy and training require reevaluation. This rigorous evaluation framework represents mature LLMOps practice, recognizing that model performance depends heavily on data accuracy ("garbage in, garbage out"). The emphasis on both alignment and quality addresses the reality that even with perfect model performance, inconsistent or unclear policies will lead to poor moderation outcomes. ## Model Optimization and Production Deployment The case study reveals several production optimization techniques essential for operating at Roblox's scale. The transition from CPU-based to GPU-based serving for high-throughput models represents a significant infrastructure investment, justified by the 4x improvement in RPS capacity. The separation of tokenization from inference is a specific architectural optimization that allows these stages to be scaled independently and potentially distributed across different hardware configurations. Model quantization and distillation are employed extensively to make large transformer-based models efficient enough for production deployment. These techniques reduce model size and computational requirements while maintaining performance, though the case study doesn't specify the precision levels used (INT8, INT4, etc.) or the distillation approaches employed (knowledge distillation from larger teacher models, task-specific distillation, etc.). The text indicates these are "essential for running a variety of multimodal models" at their scale, suggesting that without such optimizations, the infrastructure costs would be prohibitive. The emphasis on millisecond latency for text filtering reflects the user experience requirement that chat must flow naturally without perceptible delays. This constrains model complexity and requires careful optimization of the entire inference pipeline, from tokenization through model execution to result formatting and delivery. ## Operational Challenges and Continuous Adaptation The case study acknowledges significant operational challenges inherent in moderating user-generated content on a platform popular with kids, teens, and gamers who "are always introducing us to new slang terms, new trends, and new ways of evading our moderation tools." This dynamic environment requires active learning systems that continuously update models as language evolves, user patterns change, and real-world events occur. The mention of "transient examples, like slang or memes" highlights the temporal nature of some content moderation challenges—models must adapt quickly to emerging trends that may be popular for only days or weeks. The company's approach of building "systems that are scalable, fast, accurate, and consistently adapt to the dynamic world we all live in" reflects the reality that content moderation is not a solved problem requiring one-time model deployment, but rather an ongoing LLMOps challenge requiring continuous monitoring, evaluation, and improvement. The integration of human experts for "continuously improving AI, evolving and rare cases, complex investigations, and appeals" acknowledges that fully automated moderation remains insufficient for edge cases requiring nuanced judgment. The multilingual aspect adds another layer of complexity, with 28 languages supported for communication and varying levels of model performance across languages. The PII filter improvements being "rolled out across several other languages" suggests that model improvements don't automatically transfer across languages and require language-specific tuning and validation. ## Critical Assessment While Roblox presents impressive technical achievements in scale and throughput, the case study lacks certain details that would allow full assessment of their approach. Absolute performance metrics (rather than relative improvements) for precision and recall across different moderation tasks would provide clearer understanding of actual model performance. The 0.01% violation rate across 1 trillion pieces of content sounds impressive but represents millions of violations, and we don't know what percentage of actual violations this represents (i.e., the true positive rate versus missed violations). The emphasis on "AI only when it performs significantly higher in both precision and recall than humans at scale" is a reasonable principle, but the case study doesn't provide comparative benchmarks showing human versus AI performance, making it difficult to validate this claim. The statement that moderating this volume would require "hundreds of thousands of human moderators working 24/7" may be accurate but also serves to justify the AI investment without necessarily proving the AI system matches or exceeds human performance in all aspects. The open-sourcing of their voice safety classifier provides some external validation opportunity, though the case study doesn't reference independent assessments or adoption by other platforms. The real-time feedback mechanisms showing 5-6% reductions in violations represent modest but measurable improvements, suggesting these interventions have positive effects while acknowledging they're not transformative. The synthetic data generation via LLMs is presented positively for addressing rare cases, but the case study doesn't discuss potential limitations such as distribution mismatch between synthetic and real data, or how they validate that LLM-generated adversarial examples accurately represent real-world evasion attempts. The reliance on user reports and appeals for continuous improvement creates useful feedback loops but also means the system depends on user engagement and may have blind spots where violations aren't reported. Overall, Roblox demonstrates mature LLMOps practices including continuous evaluation, multiple data sampling strategies, model optimization for production deployment, real-time monitoring and feedback, and integration of human expertise for edge cases and appeals. The scale achievements are genuinely impressive, and the multi-year evolution from rule-based to transformer-based to optimized multi-model systems shows thoughtful architectural progression. However, as with any company-authored case study, claims should be considered in context of the promotional nature of the content, and the lack of independent validation or detailed performance metrics makes comprehensive assessment challenging.