## Overview This case study presents a partnership showcase between Google Cloud, NVIDIA, and ElevenLabs, demonstrating how generative voice AI workloads can be deployed at scale using GPU-accelerated Kubernetes infrastructure. The presentation, delivered at what appears to be Google Cloud Next, features speakers from all three organizations discussing the technical infrastructure that enables ElevenLabs to run their voice AI models in production. ElevenLabs is described as one of the fastest-growing AI unicorns, backed by prominent investors like Andreessen Horowitz and Sequoia. The company specializes in voice technology, including text-to-speech, voice cloning, and speech-to-speech translation. Their platform is reportedly used by 41% of Fortune 500 companies, and they generate an impressive 600 hours of audio for every hour in a day, highlighting the massive scale of their inference operations. ## The Infrastructure Foundation The presentation emphasizes the partnership between Google Cloud and NVIDIA for AI infrastructure, with both companies being recognized as leaders in the Forrester Wave for AI Infrastructure Solutions. The core infrastructure stack for running AI workloads like those of ElevenLabs consists of several layers: At the base, NVIDIA GPUs provide the accelerated compute necessary for running demanding AI workloads. These include the H100 GPUs currently available and the upcoming Blackwell B200s and GB200 NVL 72 GPUs. The presentation mentions the A3 Mega VMs announcement, which bring 2x the bandwidth of previous A3 VMs, indicating continuous infrastructure improvements for AI workloads. Google Kubernetes Engine (GKE) sits on top of this hardware layer, providing orchestration capabilities for AI deployments. The speakers emphasize that GKE removes the "heavy lifting" needed in orchestrating AI deployments, managing clusters for training, fine-tuning, and serving—all while providing portability and cost optimization. ## GPU Sharing and Optimization Strategies A significant portion of the technical discussion focuses on GPU utilization strategies, which are critical for LLMOps cost management. Three key approaches were highlighted: **Multi-Instance GPUs (MIG)**: GKE can divide a single supported GPU into seven slices, where each slice can be allocated to run one container or node while providing hardware isolation. This is particularly useful for smaller inference workloads that don't require full GPU resources. **GPU Time Sharing**: This approach enables context switching between processes running on a GPU with software isolation between workloads. This allows multiple workloads to share a single GPU, improving utilization for bursty or variable workloads. **NVIDIA Multi-Process Service (MPS)**: The presentation announces the GA launch of MPS on GKE, which enables cooperative multi-process CUDA workloads to run concurrently on a single GPU device with software isolation. This is particularly valuable for inference workloads that may not fully utilize GPU resources on their own. These GPU sharing strategies are presented as key enablers for reducing costs while maintaining performance—a critical consideration for companies like ElevenLabs that need to run inference at massive scale. ## GKE Autopilot for AI Workloads The presentation highlights GKE Autopilot's accelerator class, which allows workloads to run as a single pod on each GPU node. NVIDIA's H100 GPUs are supported on Autopilot, providing a fully managed GPU experience. This reduces operational overhead for teams running AI workloads, as they don't need to manage the underlying node infrastructure. ## NVIDIA AI Enterprise Software Stack The NVIDIA side of the partnership brings several software components designed to streamline the generative AI journey: **Nemo Toolkit**: This toolkit is designed to streamline the customization process for enterprises taking open-source models and adapting them for their use cases. It handles data curation, fine-tuning, and even the implementation of guardrails. The emphasis on being "time to market" suggests this is aimed at reducing the operational complexity of model customization. **NVIDIA NIM (Inference Microservices)**: NIM provides an easy-to-use inference microservice that aims to ensure the fastest runtime and performance for models. The presentation claims 3-4x performance improvements for models like Llama 2. Beyond raw performance, NIM is positioned as delivering significant TCO improvements, with claims that customers may need only 1/3 to 1/4 of the infrastructure for the same workload. NIM is described as portable, allowing deployment in the cloud, on-premises, or on workstations. The microservices are packaged in containers for simplified deployment. This portability is important for enterprises with hybrid deployment requirements or data sovereignty concerns. NIM is available in the GCP Marketplace with hourly pricing including support, and customers can use their GCP credits for the offering. The integration with GCP Marketplace simplifies procurement and billing for customers already in the Google Cloud ecosystem. ## ElevenLabs Production Architecture and Use Cases ElevenLabs has built a comprehensive voice AI platform running on this infrastructure. Their foundational model development followed an iterative approach: - Started with a text-to-speech foundational model - Extended the model to support multiple languages (currently 29, expanding to 40) - Added voice cloning capabilities - Created a speech-to-speech model for real-time translation The live demonstration during the presentation showcased several capabilities: voice cloning (creating a replica of a speaker's voice), text-to-speech generation with that cloned voice, and cross-lingual synthesis (translating speech to another language while maintaining the original speaker's voice characteristics). Their production products include: - **AI Dubbing Studio**: End-to-end dubbing workflow for video content - **Long-form Audio**: Used for audiobooks and podcasts - **Article Narration**: Embedded speech for articles, blogs, and websites (used by The New York Times, The New Yorker, and The Washington Post) - **Real-time Conversations**: AI assistants, chatbots, and call centers The scale of their operations is notable—600 hours of audio generated for every hour in a day—which speaks to the importance of efficient inference infrastructure. This level of throughput requires careful attention to GPU utilization, autoscaling, and cost management. ## Ethical Considerations and Guardrails ElevenLabs emphasizes consent requirements for voice cloning, stating they are "very strict about consent for voice uploading." Voices are owned by the owner of the voice, and consent is required to upload a voice. This represents an important aspect of responsible AI deployment that goes beyond pure technical operations. ## Key Takeaways for LLMOps This case study illustrates several important LLMOps patterns: The infrastructure partnership between cloud providers (Google Cloud) and hardware/software providers (NVIDIA) creates a comprehensive stack that reduces operational complexity for AI companies. The combination of managed Kubernetes (GKE), GPU optimization features (MIG, time sharing, MPS), and inference-optimized software (NIM) addresses multiple layers of the LLMOps stack. GPU utilization and cost optimization are critical concerns at scale. The emphasis on GPU sharing strategies and TCO improvements indicates that efficient resource utilization is a key operational challenge for AI inference workloads. Inference serving at scale requires careful attention to both performance and cost. The claims of 3-4x performance improvements and 1/3 to 1/4 infrastructure requirements from NIM suggest that inference optimization is a significant lever for operational efficiency. The pathway from experimentation to production is explicitly addressed, with ai.nvidia.com serving as an experimentation platform and portable NIMs enabling deployment anywhere. This supports the development-to-production workflow that is central to LLMOps practices. It's worth noting that this presentation is promotional in nature, delivered at a vendor conference with the goal of showcasing partnerships and products. The performance claims (3-4x improvements, etc.) should be validated in specific use case contexts. However, the scale metrics from ElevenLabs (600 hours of audio per hour, 41% of Fortune 500 as customers) provide concrete evidence of production deployment at significant scale.