LiftOff LLC explored deploying open-source DeepSeek-R1 models (1.5B, 7B, 8B, 16B parameters) on AWS EC2 GPU instances to evaluate their viability as alternatives to paid AI services like ChatGPT. While technically successful in deployment using Docker, Ollama, and OpenWeb UI, the operational costs significantly exceeded expectations, with a single g5g.2xlarge instance costing $414/month compared to ChatGPT Plus at $20/user/month. The experiment revealed that smaller models lacked production-quality responses, while larger models faced memory limitations, performance degradation with longer contexts, and stability issues, concluding that self-hosting isn't cost-effective at startup scale.
LiftOff LLC conducted a comprehensive evaluation of self-hosting DeepSeek-R1 language models on AWS infrastructure as a potential replacement for commercial AI services. This case study provides valuable insights into the practical challenges and economic realities of deploying open-source large language models in production environments, particularly for startup-scale operations.
The company’s primary motivation was to explore whether they could reduce dependency on paid AI coding assistants while gaining greater control over their AI infrastructure. They were particularly interested in the privacy, security, and customization benefits that self-hosting could provide, along with potential long-term cost savings.
The deployment architecture centered around AWS EC2 GPU instances, specifically testing g5g.xlarge (4 vCPU, 8 GB RAM, 16 GB GPU) and g5g.2xlarge (8 vCPU, 16 GB RAM, 16 GB GPU) configurations. The team selected the G5 series for its balance of GPU performance and pricing, along with compatibility with CUDA and machine learning frameworks.
Their deployment stack emphasized simplicity and rapid iteration. Docker served as the containerization platform, paired with NVIDIA Container Toolkit to ensure consistent environments with GPU access. This approach helped eliminate the common “it works on my machine” issues that often plague ML deployments. For model serving, they utilized Ollama, which simplified local model loading significantly, while OpenWeb UI provided a user-friendly, ChatGPT-like frontend interface suitable for internal testing by engineers.
The team conducted extensive benchmarking across four DeepSeek-R1 model variants: 1.5B, 7B, 8B, and 16B parameters. Their performance analysis revealed several critical insights about the trade-offs between model size, quality, and operational efficiency.
The smallest model (1.5B parameters) demonstrated high efficiency in terms of tokens per second and operational cost but fell short of production-quality responses. The responses lacked the sophistication and accuracy required for their intended use cases, making it unsuitable as a replacement for commercial alternatives.
Mid-tier models (7B and 8B parameters) showed noticeable improvements in response quality but still didn’t meet the threshold needed to fully replace commercial LLMs. While these models performed better than the 1.5B variant, they introduced their own operational challenges, particularly around context handling and stability under load.
The largest model tested (16B parameters) offered the best response quality but came with significant operational trade-offs. The team experienced slower throughput, dramatically higher costs, and severely limited concurrency capabilities. Performance degradation became particularly pronounced with longer contexts, with token generation speeds dropping below 30 tokens per second, which they identified as the threshold where responses begin to feel sluggish to users.
The deployment process revealed several significant technical challenges that highlight the complexity of self-hosting large language models. GPU memory limitations emerged as a primary concern, with the 16B model routinely crashing under load until the team implemented aggressive quantization techniques and heavily reduced batch sizes. Even with these optimizations, inference stability remained unreliable, especially during peak throughput periods.
Performance degradation with longer context windows posed another significant challenge. As prompt context increased, particularly with the 8B and 16B models, token generation speeds consistently dropped below their identified 30 tokens per second threshold. This led to noticeably slower response times and, in some cases, complete system crashes when the context window approached maximum limits.
The complexity of performance tuning proved more demanding than anticipated. Despite the models being pretrained, achieving optimal throughput required significant engineering effort. Minor adjustments to batch size, sequence length, or context window parameters sometimes introduced unpredictable latency or runtime crashes, requiring extensive testing and optimization cycles.
The economic analysis revealed stark differences between self-hosting and SaaS alternatives. A single g5g.2xlarge instance running 24/7 costs approximately $414 per month, while equivalent SaaS services like ChatGPT Plus cost $20 per user per month. For their target scale of under 100 internal users, the SaaS approach proved dramatically more cost-effective.
Beyond the direct infrastructure costs, the team identified significant hidden operational expenses. These included setup time, DevOps overhead, ongoing maintenance, and the engineering effort required for optimization and troubleshooting. When factoring in these additional costs, the economic gap between self-hosting and SaaS solutions widened considerably.
The cost analysis highlighted that self-hosting only begins to make economic sense at much larger scales or when specific requirements around data privacy, customization, or regulatory compliance justify the additional expense and complexity.
The experience provided the team with newfound appreciation for the engineering challenges faced by companies operating at the scale of OpenAI or DeepSeek’s hosted infrastructure. Even supporting fewer than 100 internal users pushed the limits of what they could reliably run on GPU-backed EC2 instances with the 8B and 16B models.
This realization underscored the massive infrastructure and optimization demands that grow exponentially with model size. The team recognized that delivering responsive, stable, and cost-effective LLM experiences at global scale requires sophisticated engineering capabilities that extend far beyond basic model deployment.
The case study reveals several key considerations for production readiness of self-hosted LLMs. Memory management becomes critical with larger models, requiring careful attention to quantization strategies and batch processing optimization. The team learned that concurrent request handling must be optimized from the outset, as retrofitting concurrency capabilities proved challenging.
Stability and reliability emerged as ongoing concerns, particularly with larger models under varying load conditions. The unpredictable nature of performance degradation with longer contexts suggests that production deployments would require sophisticated monitoring and failover mechanisms.
Despite the current challenges, the team maintains optimism about the future of self-hosted LLMs. They note improving GPU access and the emergence of specialized hardware like Nvidia DIGITS, which could enable developers to prototype, fine-tune, and inference reasoning AI models locally more effectively.
Their recommendation for startups and growing enterprises is clear: at smaller scales, SaaS LLM offerings currently deliver superior value for money. However, they advocate for continued experimentation and preparation for future opportunities as the technology landscape evolves.
This case study illustrates several important principles for LLMOps practitioners. The importance of comprehensive cost-benefit analysis extends beyond direct infrastructure costs to include operational overhead and engineering time. Performance benchmarking must consider not just raw throughput but also user experience thresholds and real-world usage patterns.
The experience also highlights the value of starting with simpler deployment approaches and gradually scaling complexity as requirements and capabilities mature. The team’s choice of Docker, Ollama, and OpenWeb UI created a foundation that could be incrementally improved rather than requiring wholesale architectural changes.
Finally, the case study demonstrates the importance of realistic expectations when evaluating emerging technologies. While self-hosting open-source LLMs offers compelling theoretical advantages, the practical implementation challenges and economic realities must be carefully weighed against organizational capabilities and requirements.
Cosine, a company building enterprise coding agents, faced the challenge of deploying high-performance AI systems in highly constrained environments including on-premise and air-gapped deployments where large frontier models were not viable. They developed a multi-agent architecture using specialized orchestrator and worker models, leveraging model distillation, supervised fine-tuning, preference optimization, and reinforcement fine-tuning to create smaller models that could match or exceed the performance of much larger models. The result was a 31% performance increase on the SWE-bench Freelancer benchmark, 3X latency improvement, 60% reduction in GPU footprint, and 20% fewer errors in generated code, all while operating on as few as 4 H100 GPUs and maintaining full deployment flexibility across cloud, VPC, and on-premise environments.
This panel discussion at AWS re:Invent features three companies deploying AI models in production across different industries: Somite AI using machine learning for computational biology and cellular control, Upstage developing sovereign AI with proprietary LLMs and OCR for document extraction in enterprises, and Rambler AI building vision language models for industrial task verification. All three leverage AMD GPU infrastructure (MI300 series) for training and inference, emphasizing the importance of hardware choice, open ecosystems, seamless deployment, and cost-effective scaling. The discussion highlights how smaller, domain-specific models can achieve enterprise ROI where massive frontier models failed, and explores emerging areas like physical AI, world models, and data collection for robotics.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
