Turbopuffer emerged from the challenge of building cost-effective vector search infrastructure for LLM applications when traditional solutions proved prohibitively expensive. The company developed a novel architecture leveraging S3 object storage and CPU-based processing to reduce vector storage costs by up to 95% compared to traditional in-memory solutions. Starting with a single customer (Cursor) migrating from expensive Aurora-based vector storage, Turbopuffer demonstrated that intelligent caching, clustering algorithms, and napkin-math-driven optimization could deliver performant vector search at a fraction of the cost, enabling AI companies to achieve sustainable unit economics while scaling rapidly.
Turbopuffer represents a particularly instructive case study in LLMOps infrastructure, addressing one of the most persistent challenges in production LLM deployments: the prohibitive cost of vector storage and search at scale. The company was founded in 2023 by Simon Eskildsen, a former Shopify infrastructure engineer, specifically to solve unit economics problems that were preventing AI companies from scaling their embedding-based applications.
The genesis of Turbopuffer came from direct observation of production LLM challenges. When working with Readwise, an AI-powered article recommendation service, Eskildsen built a high-quality recommendation engine using embeddings that could predict user interests with remarkable accuracy. However, when calculating the cost to run this system for all users, the infrastructure would have required $30,000 per month—a completely untenable cost for a bootstrapped company spending only $5,000 monthly on all other infrastructure combined. This stark unit economics reality led to the recommendation system being shelved entirely, a common fate for many promising LLM applications.
The fundamental challenge Turbopuffer addresses is rooted in how LLM applications typically use embeddings. When ChatGPT launched in late 2022 with extremely small context windows (4-8 kilobytes), any application wanting to connect LLMs to external documents had to implement retrieval-augmented generation (RAG) almost immediately. This meant converting documents to vector embeddings and storing them for similarity search.
Traditional vector database solutions stored embeddings primarily in DRAM (dynamic random-access memory) to achieve low-latency search. While this approach delivered excellent performance, the economics were challenging. Memory-based solutions cost approximately $2 per gigabyte, compared to S3 object storage at 2 cents per gigabyte—a 100x cost difference. For companies like Cursor, which needed to store embeddings for potentially millions of codebases, the costs quickly became unsustainable.
Eskildsen’s approach to solving this problem exemplifies first-principles engineering thinking applied to LLMOps infrastructure. Rather than accepting the conventional wisdom that vector search requires in-memory storage, he performed detailed “napkin math” calculations to determine what should theoretically be possible given hardware capabilities.
The initial architecture was deliberately minimal—what Eskildsen describes as “the simplest possible thing.” The core approach involved running a clustering algorithm on vectors to group similar embeddings together, storing clusters in files on S3 (named simply as cluster_one, cluster_two, etc.), maintaining a separate file of cluster centroids, and performing searches by downloading centroids, identifying the n closest clusters, then downloading only those cluster files.
This design ensured that all data resided durably in object storage. Even if all compute infrastructure failed, no data would be lost since every write was committed directly to S3—a critical reliability invariant for production systems. The architecture barely implemented LSM tree concepts, focusing instead on the simplest possible version that could work.
To achieve acceptable latency despite S3’s typical P99 latency of 200 milliseconds for 512KB objects, the initial system employed a reverse proxy (NGINX) as a caching layer in front of S3. This simple approach, running on a single 8-core instance, was sufficient for the initial launch. Cache invalidation was handled through shell scripts that directly manipulated NGINX’s cache directory—pragmatic engineering that prioritized shipping over perfection. Eskildsen noted he had written more NGINX Lua than most people, drawing on his Shopify infrastructure experience.
Cursor became Turbopuffer’s first customer immediately after the October 2023 launch. The timing was serendipitous—Cursor’s founders had been discussing the exact problem Turbopuffer solved, potentially even considering building it themselves. They needed a solution where actively-used codebases could reside in fast storage while inactive projects lived in cheap object storage, with seamless hot-loading between the two tiers. One of the Cursor co-founders had even tweeted about using S3 for key-value caching, demonstrating they were thinking ahead of the industry on object storage economics.
The migration from Cursor’s previous solution (a traditional vector database) to Turbopuffer delivered a 95% cost reduction on the first bill. Crucially, this wasn’t achieved through sacrificing reliability or performance, but through architectural choices that better matched the actual access patterns of LLM applications.
This case is particularly notable because it violated conventional startup wisdom. Cursor’s co-founder Sualeh Asif explicitly stated they normally would never bet their business on a tiny startup where they were the only customer—except for Turbopuffer. Several factors enabled this trust: in-person engagement (Eskildsen traveled from Canada to San Francisco to meet the team directly), demonstrable expertise (he immediately helped them debug PostgreSQL autovacuum issues with Aurora, establishing credibility), clear architectural guarantees (the S3-backed design meant data durability was never in question), and alignment on problem understanding (both teams deeply understood the unit economics challenge).
When Eskildsen showed up at Cursor’s office, they were experiencing PostgreSQL problems with AWS Aurora. He asked if they had pganalyze running and helped diagnose the issue—autovacuum hadn’t run enough, causing queries to use heap scans when they should use index scans. This database troubleshooting built immediate trust that he knew how to both operate and build databases.
The initial reverse-proxy caching approach was quickly replaced by co-founder Justine’s file-based cache implementation. This represented the beginning of continuous optimization driven by production workload understanding rather than premature optimization. Justine had been identified by Eskildsen as “the best engineer who ever worked at Shopify” and her joining validated the technical seriousness of the project.
The architecture evolved to leverage CPU-based processing with SIMD (Single Instruction, Multiple Data) instructions, particularly AVX-512, for efficient vector operations. This CPU-focused approach proved prescient as GPU scarcity became a defining constraint in the AI infrastructure landscape. By mid-2026, Turbopuffer was observing severe CPU shortages across cloud providers, driven by reinforcement learning workloads from AI labs requiring massive CPU resources for environment simulation, agent-based applications needing general-purpose compute for diverse tasks, and RL training creating feedback loops that increased CPU demand as models became more applied.
The company developed operational flexibility to handle infrastructure constraints, supporting many different instance types and SKUs across cloud providers. Favored configurations included C4 and Z4D instances on GCP, as well as ARM-based C4A instances, but the system was designed to work across diverse hardware. Eskildsen noted they work closely with cloud providers to understand which regions have available CPUs, which comes down to power availability and where new hardware is being shipped.
Turbopuffer’s development approach was shaped by an unusual relationship with venture capital. Rather than raising significant funding upfront, Eskildsen focused on a simple business constraint: Turbopuffer’s GCP bill needed to be lower than customer revenue. This “business 101” approach—foreign to many Silicon Valley companies—meant every architectural decision was evaluated through a cost-efficiency lens.
When Eskildsen first launched Turbopuffer, he was running it on a single instance in GCP, essentially blocking on whether anyone would actually care enough to use it in production. He described having worked on it all summer and being “sick of working on this”—he would only continue if there was real customer demand. This pragmatic, customer-validation-first approach is unusual for database infrastructure companies.
The initial $700K raise in January 2024 was specifically to hire two engineers (Boyan and Morgan) to accelerate learning, not to fund indefinite R&D. Boyan was someone Eskildsen had competed with at the International Olympiad in Informatics in 2012-2013, when Boyan was on the North Macedonian team and known as “God” for his programming skills. The company became profitable later that year, an unusual trajectory for database infrastructure companies. A subsequent funding round in December 2024 was explicitly for employee liquidity rather than growth capital or founder ego.
Eskildsen articulated six reasons to raise capital: to fund R&D, to fund growth, for founder ego (which he called “very dangerous”), to reward employees, for strategic partnerships, and for M&A. Turbopuffer’s first raise was reason one (R&D), and the second raise was reason four (employee liquidity). This clarity about capital purpose is refreshing compared to typical venture narratives.
Pricing was set based on napkin math of what costs should theoretically be if the system were optimally implemented, even before that optimization was complete. Cursor was promised a $4,000 monthly bill based on calculations of what efficient implementation would cost, then the engineering team worked backward to achieve that target. This reverse approach—setting price based on theoretical efficiency then building to match—is unconventional but aligned engineering work directly with customer value. Before Turbopuffer, the cheapest option was approximately $100 per million vectors; Turbopuffer launched claiming $1 per million vectors—a 100x cost reduction.
Several broader LLMOps lessons emerge from Turbopuffer’s experience:
Infrastructure cost is a first-order concern for LLM applications. The Readwise recommendation engine was technically successful but economically impossible. Many LLM features likely face similar fates—working well in demos but failing unit economics tests at scale. Infrastructure providers that reduce costs by 10-20x don’t just save money; they make entirely new application categories viable.
Access patterns matter more than raw performance specs. Cursor’s workload had natural temporal locality—developers work on a small number of active projects while most codebases sit idle. Architecture that exploited this pattern (hot cache for active projects, cold storage for inactive ones) dramatically outperformed general-purpose solutions optimized for different access patterns. The mental model is that when a developer opens a codebase, it loads into RAM within seconds, then queries are as fast as any in-memory solution, while millions of other inactive codebases sit cheaply in S3.
Simplicity enables rapid iteration. The initial implementation was deliberately minimal—no proper LSM tree implementation, basic clustering, shell script cache invalidation. This “minimum viable architecture” approach let Turbopuffer reach production quickly and learn from real workloads rather than optimizing prematurely based on assumptions. Eskildsen noted that anyone who had actually worked internally on databases would have “had too much pride to ship anything like that,” but he treated it like a SaaS project—ship the MVP, see if anyone uses it, then make it production-grade.
First principles thinking challenges industry assumptions. The conventional wisdom that vector search requires in-memory storage went largely unquestioned until someone calculated whether it was actually necessary given S3 latency characteristics, CPU capabilities, and typical query patterns. Many production LLM systems likely accept similar unexamined assumptions. Eskildsen’s “napkin math” project at Shopify—maintaining a table of fundamental performance numbers like DRAM bandwidth, S3 roundtrip costs, NVME SSD bandwidth, etc.—gave him the mental tools to reason about what should theoretically be possible.
CPU vs GPU trade-offs are evolving. While much AI infrastructure discussion focuses on GPU scarcity, Turbopuffer’s CPU-intensive approach benefited from initially abundant CPU resources. However, as RL and agent workloads proliferate, CPU scarcity is emerging as a significant constraint, forcing infrastructure teams to work closely with cloud providers on capacity planning and regional availability. Even large companies are fighting amongst each other for CPU allocations, and Turbopuffer competes for CPUs with some of the same companies they sell to.
Reliability through architectural invariants. Rather than implementing complex distributed consensus or replication protocols, Turbopuffer achieved reliability through a simple invariant: all writes commit to S3 immediately. This meant compute infrastructure could be treated as ephemeral and stateless, greatly simplifying operational complexity. If you shut down all VMs, no data is lost—the same durability guarantee the system has today.
Remote work for infrastructure teams requires intentional culture. Turbopuffer operated fully remote while building database infrastructure, traditionally considered a category requiring in-person collaboration. Their “campfire” model—encouraging but not requiring frequent in-person gatherings, with explicit incentives like turbo credits for conference talks and travel—provided flexibility while maintaining connection. Engineers could choose between frequent travel engagement or heads-down remote work based on personal preference and project needs. One engineer saw a campfire happening in New York, experienced FOMO, and took an Uber directly to the airport to fly from Ottawa to join the team—demonstrating organic engagement rather than mandated presence.
As Turbopuffer scaled beyond its initial customer, several operational challenges emerged that are instructive for LLMOps practitioners. Working with customers like Reflection AI that maximize long-term reserved capacity contracts revealed that cloud resources are not infinite even for well-capitalized companies. Turbopuffer had to collaborate with cloud providers on understanding regional power availability, upcoming hardware deployments, and capacity allocation—considerations that matter once infrastructure spend reaches meaningful scale.
Running across AWS, GCP, and Azure introduced operational complexity but also resilience and flexibility in hardware procurement. Different cloud providers had different hardware available at different times, and the ability to shift workloads across clouds became strategically valuable. Many Turbopuffer customers are AI labs and startups with deep technical expertise, creating interesting dynamics where customers understood the architectural trade-offs being made and could engage in sophisticated discussions about performance characteristics, consistency guarantees, and cost optimization strategies.
Eskildsen’s earlier work at Shopify building Toxiproxy—a proxy that could simulate database failures at the connection layer—influenced Turbopuffer’s approach to reliability. Toxiproxy was a layer 4/7 proxy that sits between applications and databases, allowing API calls to simulate failures, slowness, or other issues without mocking low-level drivers. Testing the entire stack including driver failure handling uncovered real issues in MySQL drivers and Rails that wouldn’t have been caught by mocks. This philosophy of testing realistic failure scenarios continued at Turbopuffer.
An amusing anecdote illustrates the cultural dynamics: Turbopuffer was invited to present at NVIDIA headquarters to explore partnership opportunities. Eskildsen, nervous on stage with Jensen Huang and NVIDIA leadership, joked that if things went south, Turbopuffer could pivot to vapes (a reference to the company name). Jensen responded, “Judging by your slide, maybe you should.” When Eskildsen asked “Do you vape?”, Jensen didn’t answer. Throughout the presentation, despite the team warning him not to say the “C-word” (CPUs), Eskildsen couldn’t stop talking about CPUs, AVX-512, SIMD, and how easy CPUs were to get, stopping just short of saying he was glad not to need GPUs.
The Turbopuffer case study ultimately demonstrates that production LLMOps infrastructure requires rethinking traditional database architectures through the lens of LLM-specific access patterns and cost constraints. Solutions that work well for general-purpose workloads may be economically unsustainable for embedding-heavy LLM applications, creating opportunities for purpose-built infrastructure that achieves order-of-magnitude improvements through domain-specific optimization. The success came not from sophisticated algorithms or massive capital deployment, but from deeply understanding the economics of hardware, the access patterns of LLM applications, and being willing to ship simple solutions that could be incrementally improved based on real production feedback.
Notion AI, serving over 100 million users with multiple AI features including meeting notes, enterprise search, and deep research tools, demonstrates how rigorous evaluation and observability practices are essential for scaling AI product development. The company uses Brain Trust as their evaluation platform to manage the complexity of supporting multilingual workspaces, rapid model switching, and maintaining product polish while building at the speed of AI industry innovation. Their approach emphasizes that 90% of AI development time should be spent on evaluation and observability rather than prompting, with specialized data specialists creating targeted datasets and custom LLM-as-a-judge scoring functions to ensure consistent quality across their diverse AI product suite.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Stripe, processing approximately 1.3% of global GDP, has evolved from traditional ML-based fraud detection to deploying transformer-based foundation models for payments that process every transaction in under 100ms. The company built a domain-specific foundation model treating charges as tokens and behavior sequences as context windows, ingesting tens of billions of transactions to power fraud detection, improving card-testing detection from 59% to 97% accuracy for large merchants. Stripe also launched the Agentic Commerce Protocol (ACP) jointly with OpenAI to standardize how agents discover and purchase from merchant catalogs, complemented by internal AI adoption reaching 8,500 employees daily using LLM tools, with 65-70% of engineers using AI coding assistants and achieving significant productivity gains like reducing payment method integrations from 2 months to 2 weeks.
