## Overview Roblox is a large-scale social 3D platform that enables user-generated content creation, with approximately 70 million daily active users, 300 million monthly unique users, and over 15 million user-created experiences. The platform operates across multiple device types (mobile, desktop, console, VR) with users communicating in approximately 40 different languages. This interview with Morgan McGuire, Chief Scientist at Roblox, provides extensive insight into how the company has deployed AI and LLMs in production across safety, content creation, and research initiatives. The company's AI journey spans roughly six years, beginning with safety applications and expanding into generative AI for content creation. What makes this case study particularly notable from an LLMOps perspective is the scale of deployment (processing communications for 70 million daily users), the breadth of AI applications (from text moderation to 3D content generation), and the operational philosophy of rapid iteration with weekly releases. ## Infrastructure and Architecture Roblox operates a substantial in-house technical infrastructure consisting of approximately 100,000 servers in core data centers plus 17 edge data centers distributed globally for low-latency experiences. The company employs roughly 3,000 people, with engineering and AI/ML being the fastest-growing investment areas. For their AI systems, Roblox has adopted a flexible architecture pattern where they provide a consistent Roblox interface to creators while the backend implementation can be swapped out. This architecture enables them to: - Start experimental features using third-party hosted API services to understand requirements quickly - Progressively specialize and optimize for their specific use cases - Eventually deploy completely proprietary backends optimized for their community - Switch out models on a regular basis as better options become available This approach reflects mature LLMOps thinking—decoupling the user-facing interface from the model implementation allows for continuous improvement without disrupting the user experience. ## Safety Moderation: The Foundation of Their AI Journey Roblox's first major production AI deployment was for content moderation, and it remains central to their platform. Unlike many platforms that rely on keyword filtering or reactive moderation, Roblox monitors every communication between users in real-time, aiming to maintain a positive, constructive environment. Their initial breakthrough came with adopting transformer-based models, specifically BERT and DistilBERT, for text moderation. The key LLMOps contributions from this work included: - **Performance Optimization**: They optimized these models to run approximately an order of magnitude more efficiently than the original implementations, enabling real-time processing as users type - **Multilingual Extension**: Extended originally English-only models to support 16+ languages (now approximately 40) - **Real-time Inference**: Implemented inline moderation that checks content as it's being typed, preventing inappropriate content from ever being sent The moderation system has since expanded to include: - **Voice Moderation**: Real-time speech monitoring in multiple languages with short lag, providing immediate feedback to users - **Image Moderation**: AI-based review of visual content - **Behavioral Feedback**: Rather than simply blocking content, the system provides educational feedback (e.g., "that was not the best way to phrase that") with graduated consequences This represents a sophisticated production ML pipeline handling real-time inference at massive scale with low latency requirements across multiple modalities (text, voice, images) and languages. ## Code Assist: LLM-Powered Development Tools Roblox's AI Code Assist feature has graduated from beta to full production and represents their most mature generative AI creation tool. The system is designed to help new programmers learn and accelerate development in Luau, Roblox's custom programming language. The feature works similarly to email auto-suggestions but for code, suggesting 10-15 lines of code that fit into the programmer's context. The system can recognize patterns (e.g., building a leaderboard for a game) and suggest appropriate implementations including iteration patterns, player handling, and even anti-cheating robustness. Key production metrics shared: - Over 300 million characters of code have been suggested by the tool - The system displays output character-by-character for a responsive user experience From an LLMOps perspective, several aspects are notable: **Iterative Development in Public**: Roblox explicitly tested multiple algorithms, backends, and user interfaces for this feature while in beta, gathering real user feedback to guide development. This reflects a mature approach to ML product development that embraces experimentation. **StarCoder Contribution**: Roblox collaborated on the StarCoder project, an open-source LLM specifically for code generation. Their key technical contribution was developing domain transfer techniques—training on multiple programming languages (Python, Java, etc.) to improve Lua code generation. This addressed the challenge that Luau is a less common language with limited training data compared to mainstream languages. **Hybrid Architecture**: The production system uses "a mixture of third party and proprietary solutions on the backend," with a custom Roblox frontend. This pragmatic approach allows them to leverage external capabilities while maintaining control and the ability to specialize. ## Generative AI for 3D Content Creation Roblox has released multiple generative AI tools for content creation, each addressing different parts of the 3D creation pipeline: **Material Generator**: Uses text-to-image technology adapted for physically-based material creation. Rather than generating simple colors, it produces materials with physical properties (reflectivity, roughness, translucency) that respond correctly to lighting. The initial version created tiling textures. **Texture Creator** (released recently): A more sophisticated variant that creates specialized textures for specific objects rather than generic tiling patterns. The system can identify object features (like buckles on a backpack) and apply appropriate wear patterns, dirt, and specialized materials to different parts. **Avatar Auto Setup**: Described as "probably the most powerful" creation tool, though not detailed extensively in the interview. These tools are designed with a specific philosophy: automate execution while preserving creator agency. Every output can be further edited if the creator has the skills to do so, making the tools useful for both beginners (who can ship the AI output directly) and experts (who use it as a starting point). ## ControlNet and Research Contributions A significant portion of Roblox's AI work is research-oriented, with results published and shared publicly. The ControlNet project, developed in collaboration with Stanford University (Professor Manish Agrawala), represents a major contribution to controllable generative AI. The core problem ControlNet addresses is that traditional generative AI (like image generators) offers limited control—users can only modify their text prompt and regenerate entirely. This leads to the problematic practice of "prompt engineering" with unnatural, hacker-like prompts. ControlNet's technical innovation: - Creates a small, lightweight secondary neural network that "latches onto" a pretrained model - This control network "pokes into all different levels" of the main model - Enables training in minutes rather than days/weeks on supercomputers - Dramatically reduces computational cost, time, and carbon impact - Allows natural control inputs like sketches or edge maps Practical applications include: - Preserving composition while changing content (e.g., same deer pose but in winter instead of summer) - Starting from 3D renders and applying stylization - Mixing and matching different control signals **AdaptNet**: An extension applying ControlNet principles to animation. This enables stock animations to be adapted for characters with different proportions or conditions (e.g., an injured knight holding themselves differently). ## Operational Philosophy and Deployment Practices Roblox's approach to LLMOps reflects several distinctive practices: **Weekly Release Cadence**: Unlike typical 3D software that ships every 2-4 years, Roblox deploys a new client to all users every Thursday. This applies to their AI features as well, enabling rapid iteration based on community feedback. The backend is continuously updated. **Transparent Roadmap and Public Iteration**: The company publishes its product roadmap, previews features at developer conferences, and explicitly embraces the possibility of getting things wrong initially. This philosophy—"they'd rather get it a year earlier and provide feedback"—enables faster learning cycles. **Open APIs and Open Source**: Many research contributions (StarCoder, ControlNet) are released as open source. Training methodologies and learnings are published in peer-reviewed research. **Data Advantage**: With 70 million daily users, 15 million experiences, and millions of assets, Roblox has substantial proprietary data for training and fine-tuning models. The interview explicitly notes that "whoever has the data has a real opportunity" in the AI landscape. ## Lessons Learned and Industry Observations Morgan McGuire offers several insights relevant to LLMOps practitioners: **AI Engineering vs. Model Development**: Much of the real work in AI is not creating new models but rather "learning how to prepare data, how to augment the data so that you don't need quite as much of it, learning how to normalize or regularize the data... how to prevent bias... how to prevent things like hallucination." The StarCoder papers focus primarily on methodology rather than the model itself. **Long-term Perspective**: The interview emphasizes that AI is "very early days" with "decades before the full impact is felt." This suggests organizations should invest in building sustainable AI capabilities rather than chasing short-term trends. **Control as the Key Challenge**: The next 2-3 years will focus on making generative AI controllable—how do you iterate with AI the way you would with a human collaborator? This is directly applicable to any production AI system where user intent must be preserved. **Pragmatic Technology Selection**: Using "whatever the best tool for the job is" and maintaining flexibility to swap implementations reflects practical production thinking over ideological purity. ## Conclusion Roblox presents a comprehensive example of LLMOps at scale across multiple use cases: real-time safety moderation, code generation assistance, 3D content creation, and foundational research. Their approach combines aggressive public iteration, hybrid architecture allowing backend flexibility, significant investment in optimization for their specific requirements, and a philosophy of augmenting rather than replacing human creativity. The scale (70 million daily users, 40+ languages, multiple modalities) and operational maturity (weekly releases, real-time inference requirements) make this a notable reference case for organizations deploying AI in production environments.