## Overview Clario is a prominent endpoint data solutions provider supporting clinical trials globally, with over 50 years of experience and involvement in more than 30,000 clinical trials across 100+ countries. The company developed an LLM-powered production system called the Genie AI Service to transform their software configuration process for clinical trials. This case study represents a mature implementation of generative AI in a highly regulated healthcare environment where accuracy and auditability are paramount. The use case centers on automating the creation of software configurations that manage various stages of clinical trials. Previously, this involved manually extracting data from PDF transmittal forms containing information about exams, visits, conditions, and interventions, then combining this with study details from enterprise databases and standardized exam protocols. The manual nature created risks around transcription errors, version control challenges, fragmented information flow, and extended software build timelines—all critical concerns in an industry where timing and accuracy directly impact patient outcomes. ## Technical Architecture and LLM Integration The Genie AI Service represents a comprehensive production deployment of LLMs within a clinical trials operational environment. At its core, the system uses Anthropic's Claude 3.7 Sonnet accessed through Amazon Bedrock APIs. The orchestration layer runs on Amazon ECS, providing the container-based infrastructure for reliable, scalable execution of the various microservices that comprise the solution. The architecture demonstrates thoughtful integration of LLM capabilities within existing enterprise systems. Users upload transmittal forms through a web UI over a secure AWS Direct Connect network connection, emphasizing the security requirements for handling sensitive clinical trial data. The system makes API calls to fetch study details from existing study databases, establishing the foundation for configuration by retrieving information such as study plans, participation criteria, sponsors, and collaborators. The LLM integration occurs at two primary points in the workflow. First, during data extraction, Claude Sonnet is invoked through Amazon Bedrock APIs to parse and extract structured data from PDF transmittal forms. The model identifies key fields and tables, organizes information into standardized formats, applies domain-specific rules for categorizing clinical trial visits, extracts and validates demographic fields while maintaining proper data types, handles specialized formatting rules for medical imaging parameters, and manages document-specific adaptations such as different processing for phantom versus subject scans. The second LLM integration point occurs during XML generation. After human review and validation, an internal microservice called SCSXMLConverter processes both the Software Configuration Specification document and study configurations, invoking Claude 3.7 Sonnet to generate standardized SCS XML files. These XML outputs undergo validation checks to ensure they meet the structural and content requirements of Clario's clinical study software before being released for software builds. ## Prompt Engineering and Model Optimization The case study provides valuable insights into the prompt engineering practices required for production LLM deployments in specialized domains. Clario emphasizes that few-shot prompting with domain knowledge proved foundational to their success. Rather than simple instructions, their prompts incorporate comprehensive business logic, edge case handling, and exact output formatting requirements to guide the AI's understanding of clinical trial configurations. The team discovered that prompt engineering requires substantial iteration. They spent considerable time refining prompts through multiple testing cycles, exploring different approaches to capture complex business rules about visit sequencing, demographic requirements, and field formatting. This iterative refinement process represents a critical aspect of operationalizing LLMs—the initial prompt design is merely a starting point, with continuous improvement based on real-world performance. An important challenge surfaced around data formatting variability. Transmittal forms vary significantly across therapeutic areas such as oncology and neurology, and even between studies within the same area. This variability creates difficulties when the model encounters form structures or terminology not seen during initial prompt development. Clario's approach involves continuous prompt iteration as they discover new patterns and edge cases, creating a feedback loop where human experts identify missed or misinterpreted data points that inform future prompt refinements. This represents a mature understanding of LLMOps—recognizing that production systems require ongoing prompt maintenance rather than one-time development. ## Human-in-the-Loop Validation Workflow A distinguishing aspect of this implementation is the carefully designed human-in-the-loop validation workflow. While the LLM dramatically accelerates data extraction, Clario recognized that human oversight remains essential within a structured validation workflow. The Genie AI Service provides a comprehensive review interface where stakeholders can examine AI-generated outputs, make corrections or adjustments, add comments highlighting changes as a feedback mechanism, and validate configuration accuracy. This design choice reflects practical wisdom about deploying LLMs in high-stakes environments. Rather than pursuing fully autonomous processing, which might be inappropriate given the regulatory requirements and patient safety implications of clinical trial configurations, Clario positioned the LLM as an intelligent assistant that dramatically reduces manual effort while keeping human expertise in the decision-making loop. The interface was specifically designed to highlight potential inconsistencies and provide convenient editing capabilities, enabling reviewers to apply their expertise efficiently. Validated configurations are saved to Clario's Genie Database, creating a central, authoritative, auditable source of configuration data. This centralized storage approach supports compliance requirements by maintaining complete audit trails and enabling reproducible processes—critical capabilities in the regulated clinical trials environment. ## Deployment and Infrastructure The deployment architecture demonstrates enterprise-grade infrastructure choices appropriate for production LLM applications. Amazon ECS provides the container orchestration platform, offering the scalability and reliability needed for processing variable workloads as different studies progress through configuration. The use of managed services like Amazon Bedrock reduces operational overhead by eliminating the need to manage model infrastructure directly. API integration represents a key architectural pattern throughout the solution. The system makes API calls to Amazon Bedrock for LLM inference, to study databases for retrieving study details, and between internal microservices such as the SCSXMLConverter. This API-driven architecture provides flexibility and modularity, allowing components to evolve independently while maintaining defined interfaces. Security considerations are evident in the architecture, with AWS Direct Connect providing secure network connectivity for uploading sensitive transmittal forms. The system handles protected health information and proprietary clinical trial data, requiring robust security controls throughout the data pipeline. ## Business Outcomes and Operational Impact The production deployment has delivered measurable business value across multiple dimensions. Study configuration execution time has been reduced while simultaneously improving overall quality—a dual benefit that demonstrates the system's effectiveness. By automating manual data extraction and reducing transcription errors through consistent extraction logic, the solution addresses the core pain points that motivated the project. Built-in validation safeguards help identify potential issues early in the process, preventing problems from propagating downstream into software builds and potentially impacting clinical trial operations. This shift-left approach to quality represents an important operational improvement, catching errors when they're easier and less expensive to fix. The solution has transformed collaboration patterns within Clario. Centralized review capabilities give cross-functional teams access to the same platform, improving transparency and communication efficiency. Standardized workflows have created clearer channels for information sharing and decision-making, reducing the coordination overhead that previously characterized the configuration process. From a scalability perspective, the new approach supports processing configurations across multiple studies while accommodating iterations as studies evolve. This operational flexibility is crucial in clinical trials, where protocols frequently undergo amendments and refinements. The standardized approach has also laid groundwork for expanding these capabilities to other operational areas within the organization, suggesting that the LLMOps investments are providing a foundation for broader transformation. ## Lessons Learned and LLMOps Maturity The case study provides unusually candid insights into lessons learned during the implementation, suggesting a mature approach to technology adoption. Several key learnings emerged specifically around working with generative AI technology. The team emphasizes that prompt engineering is foundational and requires iteration—recognizing that quality extraction depends heavily on well-crafted prompts encoding domain expertise. This understanding goes beyond treating LLMs as black boxes, instead viewing prompt engineering as a core competency requiring investment and ongoing refinement. The importance of human oversight within a validation workflow emerged as a critical principle. While generative AI dramatically accelerates extraction, human review remains necessary, and the system should be designed to make this review process as efficient as possible. This represents a balanced view of AI capabilities—neither dismissing LLMs as unreliable nor overestimating their readiness for fully autonomous operation in critical contexts. Integration challenges surfaced during deployment. Two-system synchronization—ensuring changes made in SCS documents are reflected in the solution—proved difficult and continues to be refined. This highlights the reality that integrating LLM-powered capabilities into existing enterprise workflows involves more than just adding AI inference—it requires rethinking data flows, system boundaries, and consistency mechanisms. The transition from proof-of-concept scripts to fully integrated solution functionality required careful planning to avoid disruption. This speaks to the operational maturity needed for production LLM deployments—moving from demonstrations to daily operational use involves addressing concerns around reliability, performance, error handling, and user training that may not be evident during initial experimentation. Process adaptation insights included the value of phased implementation, starting with pilot teams who could validate functionality and serve as internal advocates. Workflow optimization proved iterative, with the initial design evolving based on user feedback and real-world usage patterns. Even with an intuitive interface, proper training was necessary to ensure users could fully leverage the solution's capabilities. These organizational change management aspects are often underestimated in technology projects but proved essential to realizing value from the LLM deployment. Technical considerations that emerged included the need for performance optimization as processing times for larger documents initially impacted user experience, and the importance of robust error handling in the generative AI processing flow for production reliability. The variability in document formats across therapeutic areas and studies created ongoing challenges requiring continuous prompt refinement—a reminder that production LLM systems require maintenance and evolution rather than static deployment. Strategic insights from the project include the value of starting with well-defined use cases that have clear pain points and measurable outcomes, building for extensibility so the architecture can support future expansion, and tracking concrete metrics like processing time and error rates to quantify return on investment. These principles reflect a pragmatic approach to LLM adoption focused on delivering tangible business value rather than pursuing AI for its own sake. ## Assessment and Balanced Perspective This case study represents a genuinely mature production deployment of LLMs in a specialized, regulated domain. Several aspects warrant positive assessment. The human-in-the-loop design is appropriate for the high-stakes clinical trials context, recognizing that full automation may be neither feasible nor desirable given regulatory requirements and patient safety implications. The focus on specific, well-defined processes with clear pain points increased the likelihood of success compared to attempting broader, less focused AI initiatives. The technical architecture demonstrates sound engineering choices, with managed services reducing operational burden, API-driven design providing modularity and flexibility, and container orchestration on ECS enabling scalable deployment. The emphasis on auditability and compliance through centralized data storage and complete audit trails shows understanding of the regulatory environment. The candid discussion of lessons learned, including ongoing challenges around prompt iteration and system synchronization, suggests realistic expectations rather than overselling the solution's capabilities. The recognition that prompt engineering requires continuous refinement based on new document patterns demonstrates practical wisdom about operating LLMs in production. However, as with any vendor-provided case study, certain aspects merit balanced consideration. The text doesn't provide specific quantitative metrics for time reduction or error rate improvements, instead using phrases like "reduced study configuration execution time while improving quality." While directionally positive, the lack of specific numbers makes it difficult to assess the magnitude of improvements. Organizations evaluating similar implementations should seek concrete metrics during vendor discussions. The case study emphasizes the solution's benefits but provides less detail about limitations or failure modes. For instance, what happens when the LLM produces output that human reviewers don't catch during validation? What percentage of extracted data typically requires human correction? Understanding these limitations would provide a more complete picture of the system's capabilities and the level of human oversight actually required in practice. The integration challenges mentioned, particularly around two-system synchronization and ensuring changes in documents are reflected in the solution, suggest the system may not yet be fully mature. Organizations should understand that implementing similar solutions likely involves ongoing integration work rather than a one-time deployment. The reliance on a specific model (Anthropic's Claude 3.7 Sonnet) raises questions about model flexibility and vendor lock-in. While Amazon Bedrock provides access to multiple models, the extensive prompt engineering investment for Claude creates switching costs if another model becomes more suitable. Organizations should consider whether their architecture supports model flexibility or commits them to specific vendors. Overall, this case study demonstrates that production LLM deployments in specialized domains are feasible and can deliver meaningful business value when approached thoughtfully. The combination of appropriate use case selection, sound technical architecture, realistic human-in-the-loop design, and honest assessment of ongoing challenges positions this as a valuable reference for organizations considering similar implementations. The emphasis on prompt engineering as an iterative discipline requiring domain expertise, continuous refinement, and organizational investment represents an important maturity indicator that other organizations would do well to emulate.