## Overview DDI is a global leadership development and assessment company that has been serving clients across various industries for over 50 years, reaching more than 3 million leaders annually. The company specializes in behavioral simulations designed to evaluate decision-making, problem-solving, and interpersonal skills in leadership candidates. This case study examines how DDI transitioned from a manual, human-assessor-based evaluation process to an automated LLM-powered solution, significantly reducing turnaround times while maintaining scoring accuracy. The core business problem centered on the inefficiency of the existing assessment workflow. Candidates would complete behavioral simulations, and trained human assessors would then evaluate their responses. This manual process, which involved thorough analysis and score input into their systems, typically took 24 to 48 hours. This delay created friction in the customer experience and limited DDI's ability to scale their operations cost-effectively. ## Technical Challenges and Infrastructure Requirements Before adopting their current solution, DDI faced several operational challenges that are common in enterprise LLMOps deployments. These included hardware orchestration and infrastructure management for ML workloads, scaling issues for both exploration and model training phases, ensuring data privacy and security for sensitive assessment data, managing operational costs effectively, and the complexity of coordinating with multiple vendors to achieve end-to-end solutions. Previous experiments with GenAI workloads including prompt engineering, RAG, and fine-tuning had been conducted, but DDI needed a more comprehensive, integrated solution. The team chose Databricks as their unified platform partner, which provided what they described as a "dual advantage": the ability to build models and a repository to serve them from. This consolidation proved crucial for simplifying their AI development and deployment processes. ## LLM Development and Experimentation Approach DDI's approach to developing their automated assessment system involved a methodical exploration of various prompt engineering and model optimization techniques. The team began experimenting with OpenAI's GPT-4 as their initial baseline, testing several prompting strategies to understand what would work best for their specific use case. The prompt engineering techniques explored included few-shot learning, which allowed them to quickly adapt models to different types of behavioral simulations by providing limited examples. They also implemented chain-of-thought (COT) prompting, which structures prompts to mimic human reasoning by breaking down complex assessment problems into smaller, manageable steps. Additionally, they experimented with self-ask prompts that enabled the model to generate its own questions and answers to better understand and process the simulation responses. The experimentation was facilitated through Databricks Notebooks, which provided an interactive, web-based interface for writing and executing code, visualizing data, and sharing insights. According to the case study, this enabled a highly collaborative environment where experimentation became the norm. The notebooks were used to orchestrate prompt optimization and instruction fine-tuning for LLMs at scale, with the benefit of managed infrastructure support. ## Prompt Optimization with DSPy A significant technical achievement in this project was the use of DSPy for prompt optimization. DSPy is a framework that allows for programmatic and optimized prompting rather than relying solely on manual prompt engineering. The results were substantial: the recall score improved from 0.43 to 0.98, representing more than a 100% increase in this metric. This improvement is particularly noteworthy because recall is critical in assessment contexts where missing positive cases (false negatives) could mean failing to identify candidates who should have scored well on particular competencies. It's worth noting that while the recall improvement is impressive, the case study doesn't provide detailed information about potential tradeoffs, such as whether precision was affected. In classification tasks, dramatic improvements in recall can sometimes come at the cost of increased false positives. However, the overall F1 score improvements reported later suggest that the system achieved a reasonable balance. ## Model Fine-Tuning with Mosaic AI Model Training Beyond prompt optimization, DDI employed instruction fine-tuning using Llama3-8b as their base model. The instruction fine-tuned model achieved an F1 score of 0.86, compared to a baseline score of 0.76, representing a 13% improvement. This fine-tuning approach allowed DDI to adapt an open-source foundation model to their specific domain of behavioral assessment without the costs and complexity of training a model from scratch. The use of an 8-billion parameter model like Llama3-8b represents a pragmatic choice that balances capability with operational costs and latency requirements. For a production system that needs to deliver results in seconds rather than minutes, smaller, optimized models often outperform larger models in terms of cost-efficiency while still meeting accuracy requirements. ## MLOps Lifecycle Management The case study highlights the use of MLflow, an open-source platform developed by Databricks, for managing the LLM operations lifecycle. MLflow was used for tracking experiments across their various prompt engineering and fine-tuning approaches, logging artifacts as `pyfunc` models (Python function models) which provides flexibility in deployment, tracing LLM applications for debugging and monitoring, and automated GenAI evaluation to systematically assess model performance. The integration with Unity Catalog provided unified governance capabilities including fine-grained access controls, centralized metadata management, and data lineage tracking. This is particularly important for an organization like DDI that handles sensitive assessment data and needs to maintain clear audit trails. ## Production Deployment Architecture The deployment architecture leveraged several key components of the Databricks platform. Models were registered to Unity Catalog and deployed as endpoints with auto-scaling and serverless computing capabilities. This serverless approach is noteworthy because it allows DDI to handle variable workloads without maintaining dedicated infrastructure, reducing costs during low-usage periods while automatically scaling up when demand increases. Security integration was achieved through Azure Active Directory (AAD) groups synced with the Databricks account console via a SCIM provisioner. This ensures that access controls from their enterprise identity system are properly reflected in the data platform, maintaining consistent security policies across the organization. ## Results and Business Impact The automated system reduced simulation report delivery time from 48 hours to just 10 seconds, an improvement of approximately four orders of magnitude. This dramatic reduction in turnaround time has multiple business implications: leaders receive immediate feedback which improves the customer experience, operational costs are reduced by eliminating the need for human assessors for routine evaluations, and DDI can scale their operations more effectively. The case study claims that the LLMs have demonstrated high reliability and precision in their scoring, though it's important to note that this is a vendor-published case study and independent verification of these claims isn't provided. The metrics reported (0.86 F1 score for the fine-tuned model) are respectable for a classification task but not perfect, suggesting there are likely some assessments where the automated system may differ from what a human assessor would have determined. ## Future Directions DDI has indicated plans to further enhance their models through continued pretraining (CPT) using Mosaic AI Model Training. This approach involves training models on domain-specific language from their proprietary corpus to incorporate internal knowledge about leadership assessment and behavioral evaluation. The resulting base model, trained with DDI-specific data, could then be further fine-tuned for various simulation analysis use cases. This planned progression from prompt engineering to instruction fine-tuning to continued pretraining represents a common maturation path in enterprise LLM adoption. As organizations become more sophisticated in their AI capabilities and accumulate more domain-specific data, they often move toward more customized model training approaches that better capture their unique domain knowledge. ## Critical Assessment While the case study presents impressive results, several factors warrant balanced consideration. The content is published by Databricks as a customer success story, so it naturally emphasizes positive outcomes. The specific metrics reported are point-in-time measurements, and production performance over time with diverse inputs may vary. The comparison to human assessors focuses on speed; a more complete picture would include detailed accuracy comparisons across different types of assessments. Nevertheless, the case study demonstrates a well-structured approach to LLMOps that includes systematic experimentation with multiple prompting techniques, use of modern frameworks like DSPy for programmatic prompt optimization, leveraging fine-tuning to customize open-source models, proper governance and security integration, and serverless deployment for cost-effective scaling. This represents a mature implementation pattern that other organizations considering similar LLM-powered automation projects could learn from.