## Overview Deepgram, a mid-stage speech-to-text startup founded in 2015 with $85 million in Series B funding, presents a compelling case for using small, domain-specific language models (SLMs) in production environments rather than relying on large foundational language models. The presentation, given by Andrew (VP of Research at Deepgram), focuses specifically on the call center industry as a prime example of where domain adaptation provides significant advantages over general-purpose LLMs. Deepgram claims to have processed over one trillion minutes of audio through their platform, giving them substantial experience in production speech AI systems. The core thesis of this case study is that businesses will derive maximum benefit from language AI products that are cost-effective, reliable, and accurate—all three together—and that achieving this trifecta requires small, fast, domain-specific language models rather than large foundational models. This represents an important counterpoint to the prevailing industry trend of using ever-larger models for all applications. ## The Multi-Model Pipeline Architecture Deepgram proposes a three-stage language AI pipeline that reflects real-world production requirements for speech intelligence applications: **Perception Layer**: This foundational layer consists of Automatic Speech Recognition (ASR) that converts audio to text, complemented by a diarization system that identifies who is speaking when. The diarization component is crucial for formatting transcripts in a way that separates speaker turns, which is essential for downstream understanding tasks in conversational contexts. **Understanding Layer**: This layer takes the transcript output and applies language models (whether large, medium, or small) to extract meaningful insights. The key outputs include summarization of transcripts, topic detection, and sentiment analysis. Deepgram argues this is where domain-specific models provide the most value. **Interaction Layer**: For applications requiring response generation, this layer takes LLM output and converts it back to audio using text-to-speech systems, enabling voice bots and real-time agent assist systems. This architecture represents a practical approach to production deployment where each component can be optimized independently while working together in an integrated pipeline. ## The Problem with Large Language Models in Production The presentation makes several pointed arguments against using prompted foundational LLMs for domain-specific applications like call centers: **Scale and Efficiency Issues**: Large language models with 100 billion+ parameters require enormous computational resources. The speaker notes that launching such models would require dozens of clusters with 5,000 GPUs each. Inference times are on the order of 100 milliseconds per token, meaning generating longer responses can take many seconds—unacceptable for real-time applications like agent assist systems. **Domain Mismatch**: While LLMs have broad general knowledge and can be aligned to many tasks without explicit training, conversational text from call centers has a high degree of specificity. Call center conversations cover narrowly distributed topics, feature unique speech patterns based on geographic location, and contain a long tail of rare words and industry-specific terminology that foundational models likely haven't encountered during training. **Unrealistic Text Generation**: Perhaps most compellingly, the presentation demonstrates that when prompted to continue a call center conversation, ChatGPT generates highly unrealistic text. The generated speech is "way too clean as if it were written" and follows predictable scripts (greeting, issue description, agent action, customer acceptance, call end). Real call center transcripts, in contrast, feature crosstalk (people talking over each other), disfluencies (stuttering, stumbling, filler words), and other messy characteristics that make them difficult to read and interpret. This distribution mismatch means foundational LLMs are fundamentally ill-suited for understanding and generating realistic call center content. ## The Domain-Specific SLM Solution Deepgram's solution involves transfer learning a relatively small language model (500 million parameters) that was initially trained on general internet data. They then fine-tuned this model on an in-domain dataset of call center transcripts. It's worth noting that while 500M parameters is small compared to models like GPT-4, it's still a substantial model that requires careful deployment considerations. **Training Results**: The domain-adapted model showed dramatic improvements across all metrics, including next-token prediction loss and perplexity. When prompted to continue a call center conversation, the adapted model generates "very realistic text" that reflects the actual characteristics of call center discourse, including the messiness and unpredictability absent from foundational model outputs. **Summarization Capability**: Building on the domain-adapted base model, Deepgram trained a specialized summarization model for call center transcripts. This represents a practical application of the transfer learning approach where domain knowledge is first captured in the base model and then task-specific capabilities are layered on top. ## Production Demonstration The presentation includes a live demonstration using a Jupyter notebook that showcases the end-to-end system. The workflow involves three key steps: The system calls a function that hits the Deepgram API, which transcribes an audio phone call and diarizes the transcript. This combined output is then sent to their SLM for summarization. The entire process—transcription, diarization, and summarization—completed in 8.4 seconds for the demonstrated call. The results showed a "highly accurate" transcript with proper speaker separation, and a "very nice human readable summary" that accurately described what happened in the call. This sub-10-second processing time is crucial for practical applications where near-real-time insights are needed. ## LLMOps Considerations and Implications From an LLMOps perspective, this case study raises several important considerations for practitioners: **Model Size vs. Domain Adaptation Trade-off**: The presentation makes a strong case that domain adaptation of smaller models can outperform larger general-purpose models for specific applications. This has significant implications for infrastructure costs, latency requirements, and deployment complexity. A 500M parameter model is dramatically easier to deploy and scale than a 100B+ parameter model. **Multi-Model Pipeline Complexity**: The three-stage architecture (perception, understanding, interaction) introduces operational complexity as multiple models must be deployed, monitored, and maintained. However, this modular approach also enables independent optimization of each component and allows for targeted improvements without system-wide retraining. **Data Requirements for Domain Adaptation**: The approach requires access to substantial in-domain training data (call center transcripts in this case). Organizations considering this approach need to evaluate whether they have sufficient domain-specific data and the rights to use it for model training. **Latency Considerations**: The emphasis on speed (8.4 seconds for the full pipeline, arguments about 100ms/token being too slow for LLMs) highlights the critical importance of latency in production speech AI systems. Call center applications often require real-time or near-real-time processing for agent assist systems. **API-First Architecture**: Deepgram's approach of exposing these capabilities through an API aligns with modern MLOps best practices, enabling customers to integrate speech intelligence without managing the underlying infrastructure. ## Critical Assessment While the presentation makes compelling arguments, several caveats should be noted. The comparison with ChatGPT's call center text generation, while illustrative, may not represent the best that modern LLMs can achieve with better prompting or few-shot examples. Additionally, the specific metrics and performance numbers for the domain-adapted model are mentioned as "dramatically improved" but specific values aren't provided, making it difficult to quantify the actual gains. The claim that foundational LLMs are fundamentally unsuited for this domain may also be somewhat overstated—more recent models with better instruction following and larger context windows might perform better on these tasks than the presentation suggests. However, the cost and latency arguments remain valid concerns for production deployments. Furthermore, while Deepgram claims to have the "best" and "most accurate" speech-to-text API on the market, these claims should be evaluated independently against competitors. The company has clear commercial interests in promoting their approach over alternatives. ## Conclusion This case study represents an important perspective in the ongoing discussion about how to best deploy language AI in production. Rather than defaulting to the largest available models, Deepgram advocates for a more nuanced approach where domain-specific smaller models can provide better cost-effectiveness, reliability, and accuracy for specific applications. The call center use case serves as a concrete example where the unique characteristics of the domain—messy transcripts, specific terminology, real-time requirements—make domain adaptation particularly valuable. For LLMOps practitioners, this presents a valuable framework for evaluating when smaller, domain-adapted models might be preferable to large foundational models.