## Overview Vannevar Labs is a defense-tech startup that provides advanced software and hardware solutions to support the U.S. Department of Defense (DoD) in deterring and de-escalating global conflicts, particularly with Russia and China. The company serves hundreds of mission-focused users across various DoD branches, with use cases ranging from maritime sensing systems to sentiment analysis for tracking misinformation. This case study focuses on their journey to build a production-ready sentiment analysis system capable of classifying the sentiment of news articles, blogs, and social media content related to specific narratives—a critical capability for understanding the strategic communications of nation-states. The case study is published by Databricks and naturally emphasizes the benefits of their Mosaic AI platform. While the reported results are impressive, readers should note that this is vendor-published content and independent verification of the specific metrics would strengthen these claims. ## The Problem: Limitations of Commercial LLMs Vannevar Labs initially attempted to use GPT-4 with prompt engineering for their sentiment analysis needs. However, this approach presented several significant challenges that are commonly faced when using commercial LLMs in specialized production environments: **Accuracy Limitations**: The best accuracy the team could achieve with GPT-4 was approximately 65%, which was insufficient for their mission-critical defense applications. This highlights a common LLMOps challenge where general-purpose models, despite their broad capabilities, often underperform on domain-specific tasks without customization. **Cost Constraints**: Running inference through GPT-4's API proved too expensive for Vannevar's operational requirements, especially when processing large volumes of multilingual content. This is a recurring theme in production LLM deployments where API-based commercial models can become cost-prohibitive at scale. **Multilingual Performance Issues**: Vannevar's data spans multiple languages including Tagalog, Spanish, Russian, and Mandarin. GPT-4 struggled particularly with lower-resourced languages like Tagalog, demonstrating how even state-of-the-art commercial models may have gaps in multilingual capabilities, especially for less common languages. **Infrastructure Challenges**: The team faced difficulties in spinning up GPU resources to fine-tune alternative models, as GPUs were in short supply at the time. This reflects broader industry challenges around compute resource availability that can bottleneck LLMOps initiatives. **Label Collection**: Gathering sufficient instruction labels to fine-tune models was described as a company-wide challenge, highlighting the often-underestimated effort required for data preparation in supervised fine-tuning workflows. ## The Solution: Databricks Mosaic AI To overcome these hurdles, Vannevar Labs partnered with Databricks and leveraged the Mosaic AI platform to build an end-to-end compound AI system. The solution architecture encompassed data ingestion, model fine-tuning, and deployment. ### Model Selection and Fine-Tuning The team chose to fine-tune Mistral's 7B parameter model for several strategic reasons: - **Open Source Nature**: Using an open-source model provided flexibility in deployment and customization without vendor lock-in concerns - **Efficient Hardware Requirements**: The model could operate on a single NVIDIA A10 Tensor Core GPU, meeting their real-time latency requirements while minimizing infrastructure costs - **Domain Adaptation**: Fine-tuning with domain-specific data allowed the model to learn patterns specific to defense intelligence sentiment classification The fine-tuning process utilized Mosaic AI Model Training, which provided the infrastructure for efficient training across multiple GPUs when needed. The team followed a comprehensive workflow that included MDS (Mosaic Data Streaming) conversion, domain adaptation, instruction fine-tuning, and model conversion for deployment. ### Infrastructure and Orchestration A critical component of the LLMOps success was the orchestration tooling provided by Mosaic AI: **MCLI (Mosaic Command Line Interface) and Python SDK**: These tools simplified the orchestration, scaling, and monitoring of GPU nodes and container images used in model training and deployment. The MCLI's capabilities for data ingestion allowed secure, seamless connection to Vannevar's datasets, which was crucial for the model training lifecycle. **YAML-Based Configuration Management**: Databricks facilitated efficient training across multiple GPUs by managing configurations through YAML files. This approach significantly simplified orchestration and infrastructure management, allowing the team to easily adapt training parameters without extensive code changes. **Third-Party Tool Integration**: The platform integrated with monitoring tools like Weights & Biases, enabling comprehensive experiment tracking and model performance monitoring throughout the training process. ### Model Export and Deployment The platform enabled Vannevar Labs to convert their trained models to a standard Hugging Face format and export them to their Amazon S3 or Hugging Face Model Repository for production use. The team benefited from example repositories provided by Databricks that outlined the complete workflow, which they adapted for their specific use case. The Senior ML Engineer at Vannevar Labs specifically praised a Hugging Face repository that provided comprehensive examples of the full workflow for fine-tuning LLMs from scratch, primarily using the MPT-7B model as a reference. This knowledge transfer accelerated their development timeline significantly. ## Results and Production Impact The implementation delivered measurable improvements across multiple dimensions: **Accuracy Improvement**: The fine-tuned model achieved an F1 score of 76%, representing an 11-percentage-point improvement over the 65% accuracy achieved with GPT-4. While a 76% F1 score may still leave room for improvement in high-stakes applications, the relative gain is substantial. **Latency Reduction**: Inference latency was reduced by 75% compared to previous implementations. This dramatic improvement enabled the team to run large backfill jobs and process significantly more data efficiently, which is critical for real-time defense intelligence applications. **Cost Efficiency**: The solution proved more cost-effective than the GPT-4 approach, though specific cost figures were not disclosed. The ability to run on a single A10 GPU suggests infrastructure costs were kept manageable. **Rapid Deployment**: The entire process—from initial tutorial exploration to deploying a fully functional, fine-tuned sentiment analysis model—took approximately two weeks. This rapid deployment timeline demonstrates the value of managed LLMOps platforms in accelerating time-to-production. ## LLMOps Best Practices Highlighted This case study illustrates several important LLMOps principles: **Knowing When to Fine-Tune**: When commercial models fail to meet accuracy, cost, or latency requirements, fine-tuning smaller, domain-specific models can be a more effective approach. This case demonstrates the classic trade-off between general-purpose large models and specialized smaller models. **Infrastructure Abstraction**: Using managed platforms that abstract GPU provisioning and orchestration complexity allowed the team to focus on model development rather than infrastructure management. **Standard Model Formats**: Converting to Hugging Face format for deployment ensured portability and compatibility with standard inference tooling. **Monitoring Integration**: Integration with tools like Weights & Biases from the start enabled proper experiment tracking and production monitoring. **Multilingual Model Training**: Fine-tuning on domain-specific multilingual data addressed the lower-resourced language gaps that commercial models exhibited. ## Considerations and Caveats While the results are impressive, several factors warrant consideration: - The case study is published by Databricks, the platform vendor, so readers should approach the metrics with appropriate skepticism - Specific details on the volume of training data, annotation process, and evaluation methodology are not disclosed - The 76% F1 score, while improved, may still require human review for high-stakes intelligence applications - Long-term maintenance, model drift monitoring, and retraining strategies are not discussed Overall, this case study demonstrates a practical approach to moving from prompt engineering with commercial APIs to fine-tuned domain-specific models when the former fails to meet production requirements. The emphasis on rapid deployment, managed infrastructure, and measurable improvements across accuracy, latency, and cost provides a useful template for organizations facing similar LLMOps challenges.