## Overview This case study documents an early-stage prototype application called DesignBench, developed by Dan Becker through Build Great AI, which demonstrates a novel approach to bridging the gap between natural language and physical object creation. The application emerged from observations made while teaching LLM fine-tuning courses to thousands of students, where Dan noticed that despite high enthusiasm for AI, few participants had concrete ideas for useful products. This inspired the focus on creating tangible, physical outputs—what Dan describes as moving "from bits to atoms." The core premise is democratizing 3D design: most people who own 3D printers (estimated at 90%) don't know what to do with them, largely because CAD software has a steep learning curve. DesignBench allows users to describe objects in natural language and receive 3D-printable designs within minutes. ## Technical Architecture and Multi-Model Strategy One of the most interesting LLMOps aspects of this case study is the deliberate use of multiple LLMs in parallel rather than relying on a single model. The system simultaneously queries several models including: - GPT-4o (OpenAI) - Claude Sonnet 3.5 (Anthropic) - Llama 3.1 70B (Meta, served via Groq) This multi-model approach is a pragmatic response to the current limitations of LLMs in spatial reasoning. As Dan explicitly acknowledges, spatial awareness is "really bad" for LLMs as of August 2024. Many generated objects have detached parts, incorrect proportions, or other fundamental issues. By running multiple models simultaneously, the application provides users with a variety of outputs—some will inevitably be poor, but others may be closer to what the user wants. The system also experiments with different prompting strategies for each model, including Chain of Thought versus direct prompting. This creates a matrix of outputs: multiple models × multiple prompting strategies × multiple CAD languages. The philosophy is that in the face of uncertainty about what will work best, breadth of experimentation compensates for individual model limitations. ## Code Generation and CAD Languages The LLMs don't generate 3D models directly—they generate code in CAD languages, primarily OpenSCAD. This code is then rendered to produce the visual 3D model and can be exported as STL files, the standard format for 3D printing software. This approach leverages the strength of LLMs in code generation while outsourcing the actual rendering to deterministic CAD software. The choice of OpenSCAD as a target language is notable because it's a programmatic CAD language where objects are defined through code rather than visual manipulation. This makes it more suitable for LLM generation than GUI-based CAD tools. The system experimented with multiple CAD languages, though OpenSCAD appears to be a primary target. One advantage of generating code rather than direct 3D representations is that the code can be inspected, debugged, and manually modified if needed. Users can view the generated code through a "get code" option in the interface. ## Inference Infrastructure and Latency Considerations An interesting operational detail is the use of Groq for serving Llama 3.1 models. Dan specifically mentions that when watching the application populate results, "you always have one or two that pop up way before the others"—those are the Groq-served Llama models. This highlights an important LLMOps consideration: when running multiple models in parallel for user-facing applications, inference latency varies significantly across providers. The choice of Groq was partly practical—at the time of recording, it was free, though Dan expressed hope for a paid account to allow more aggressive usage. This reflects the reality of early-stage projects navigating the evolving pricing and availability landscape of LLM inference providers. Regarding model quality versus speed trade-offs, Dan noted that the Llama 3.1 70B model (the largest he was using via Groq) and GPT-4o Mini are "really not very good" compared to GPT-4o and Claude Sonnet 3.5. However, the speed advantage of Groq-served Llama makes it valuable in a multi-model setup where users benefit from fast initial results while waiting for higher-quality models to complete. Dan mentioned that the 400B parameter Llama model (supposedly competitive with GPT-4o) hadn't been tested yet, suggesting potential for quality improvements with larger open models. ## Iterative Design Through Conversation A key UX and LLMOps pattern demonstrated is iterative refinement through conversation. Users don't expect perfect results from the first prompt—instead, they select a promising design from the initial batch and then refine it through follow-up prompts. Examples from the demo include: - Starting with "a cup with the name Hugo engraved in it" - Refining with "the name Hugo seems to be floating on the inside of the cup, have it be letters raised from the bottom inside of the cup and make the letters be pretty tall" - For a dog design: "make the legs longer" and "make it a Great Dane" This mirrors the natural workflow in traditional CAD software where designers iterate, but with natural language as the interface. The key insight is that imperfect initial results are acceptable when iteration is fast and intuitive. ## Multimodal Capabilities and Future Directions The application includes an "upload image" feature, leveraging the multimodal capabilities of modern LLMs. Dan describes a use case from a neighbor who is a hobbyist inventor: the ideal workflow would be to sketch a design on paper and show that image to the model rather than describing it in text. This represents an interesting extension of the text-to-3D paradigm. Several future directions are mentioned that relate to LLMOps practices: - **Fine-tuning**: Not currently implemented, but identified as a potential future improvement. This would involve training models specifically on CAD generation tasks. - **RAG for examples**: Rather than static few-shot examples in the prompt, dynamic retrieval of relevant examples based on the user's request. - **More examples in prompts**: Even without RAG, increasing the number of examples in prompts could improve output quality. - **Tool calling for functional design**: An ambitious goal to enable the LLM to call finite element analysis tools, allowing for functional specifications (e.g., "a catapult that can shoot a checker X meters when printed with this type of plastic"). ## Honest Assessment of Limitations One refreshing aspect of this case study is the candid acknowledgment of limitations. Dan repeatedly emphasizes that spatial reasoning is challenging for current LLMs, many generated objects are unusable, and the complexity ceiling is lower than professional CAD software. The application is explicitly positioned as useful for "the home inventor who's going to make something small" rather than for professional architects or engineers designing buildings. This honesty about scope is valuable from an LLMOps perspective—setting appropriate user expectations is crucial for adoption and satisfaction. ## Practical Results The demonstration showed tangible results: a personalized cup design was refined from initial prompt to final STL file in a few minutes of conversation, and Dan subsequently 3D printed and sent photos of the physical cup to the podcast host. The estimated time savings compared to traditional CAD software was dramatic—what might take hours even for someone experienced with CAD software was accomplished in minutes. ## Early Stage Considerations This is explicitly described as a "pre-Alpha" hobby side project started less than a month before the recording. There's no monetization at this stage, and Dan is actively seeking beta testers with 3D printers. The website DesignBench.ai was mentioned as the hosting location. From an LLMOps maturity perspective, this represents the experimental/prototype phase where the focus is on demonstrating feasibility and gathering user feedback rather than production-scale concerns like reliability, monitoring, or cost optimization at scale. However, the architectural decisions—multi-model orchestration, code generation as an intermediate representation, and iterative refinement—represent patterns that would carry forward into a production system.