## Overview AstraZeneca, a global biopharmaceutical company, implemented a comprehensive AI initiative called "Development Assistant" - an interactive AI agent designed to accelerate drug development processes. The company, which has set ambitious goals to deliver 20 new medicines by 2030 while becoming an $80 billion company and achieving carbon negativity, recognized that traditional approaches to drug development needed augmentation with AI capabilities to meet these targets. The Development Assistant represents a strategic shift from isolated AI proof-of-concepts to a production-ready, scalable AI system that serves multiple R&D domains. The initiative was led by Rashali Goyle, Senior Director within R&D IT at AstraZeneca, and demonstrates a mature approach to LLMOps that encompasses technical architecture, data governance, and organizational change management. ## Technical Architecture and Evolution ### Initial Single-Agent Approach The Development Assistant began as a single-agent system built on top of AstraZeneca's existing 3DP (Drug Development Data Platform) infrastructure. This platform was strategically selected because it already contained well-structured data products based on various clinical, regulatory, and safety systems. The choice to build on existing infrastructure was deliberate - the 3DP platform already had thousands of users, established security and privacy controls, and provided "AI-ready" data products that followed FAIR (Findable, Accessible, Interoperable, Reusable) principles. The initial implementation demonstrated impressive rapid development capabilities, with a working prototype created in just one week. The system enabled natural language querying of clinical trial data, automatically converting user questions into SQL queries and providing visualizations. For example, a user could ask "What are the top five countries with the most clinical trial sites?" and receive both the reasoning steps, the generated SQL query, and a pie chart visualization of the results. This transparency in reasoning was a deliberate design choice to enable quality checks and verification. Users could see exactly which data tables were being queried and how the LLM interpreted their request, allowing them to validate the appropriateness of the response and trace back to source data when needed. ### Challenges with Single-Agent Architecture Despite the initial success, AstraZeneca quickly identified limitations with the single-agent approach when attempting to scale across multiple R&D domains. The team discovered several critical issues: **Performance Bottlenecks**: As the volume of data increased across clinical, regulatory, quality, and other R&D domains, the single agent began experiencing significant performance degradation. The system became slower and less responsive when handling complex queries across large datasets. **Increased Hallucinations**: With more diverse data sources and domain-specific terminology, the single agent began producing less accurate responses. The team observed more hallucinations and less effective outputs as the system attempted to handle the full breadth of pharmaceutical R&D data. **Complex Problem Solving Limitations**: The single-agent architecture proved insufficient for handling complex, multi-domain problems that required specialized knowledge across different areas of drug development. **Domain-Specific Terminology Conflicts**: A critical challenge emerged around controlled vocabularies and acronyms. The pharmaceutical industry relies heavily on domain-specific terminology, and the team found that terms could have different meanings across domains. For example, certain acronyms might refer to different concepts in clinical trials versus regulatory submissions. ### Multi-Agent Architecture with Amazon Bedrock To address these limitations, AstraZeneca evolved to a multi-agent architecture using Amazon Bedrock. This approach introduced a supervisor agent that coordinates with specialized sub-agents, providing the flexibility and scalability that was lacking in the single-agent system. The multi-agent architecture works as follows: - A **supervisor agent** receives user prompts and determines the appropriate routing - **Terminology agents** handle domain-specific vocabulary and acronym disambiguation - **Domain-specific agents** specialize in particular areas like clinical trials, regulatory affairs, or quality assurance - The supervisor orchestrates the workflow between agents to provide comprehensive responses This architecture enables cross-domain value chain benefits while maintaining specialized expertise in each area. When a clinical user submits a query, the supervisor agent can route it through the terminology agent first to ensure proper interpretation, then direct it to the appropriate domain-specific agents for processing. ## Data Management and Augmentation Strategies ### Controlled Vocabulary Integration One of the key technical insights from AstraZeneca's implementation was the critical importance of augmenting LLMs with appropriate controlled vocabularies. The pharmaceutical industry's heavy reliance on acronyms and specialized terminology created significant challenges for accurate query interpretation. The team implemented controlled vocabulary augmentation to help the LLM understand domain-specific terms correctly. For instance, when dealing with medical conditions like "lymphoid leukemia," the system needed access to proper medical terminology to generate robust outcomes. This augmentation process involved integrating existing company vocabularies and domain-specific terminologies into the LLM's context. ### Metadata Enhancement Another significant technical challenge was the quality of metadata column labeling across different data sources. The team acknowledged being "pretty lazy with our metadata column labeling at times," which affected the LLM's ability to generate appropriate responses to queries. They implemented metadata augmentation processes to improve the LLM's understanding of data structure and content, enabling more accurate query generation and response formulation. ### Data Product Strategy The decision to build on the 3DP data platform's data products rather than raw data sources was strategically important for LLMOps success. Data products provided several advantages: - **Standardized Structure**: Data products followed consistent schemas and quality standards - **Business Logic Integration**: Domain expertise was already embedded in the data product definitions - **Scalability**: New data sources could be onboarded through the data product framework - **Governance**: Security, privacy, and compliance controls were already established ## Production Deployment and Operations ### Rapid Time to Production AstraZeneca achieved production deployment of their Development Assistant MVP within six months from concept initiation. This rapid timeline was enabled by several factors: - Building on existing infrastructure (3DP platform) - Leveraging established security and privacy controls - Using proven data products rather than raw data sources - Strong collaboration with AWS technical teams The six-month timeline stands out in the pharmaceutical industry, where regulatory requirements and risk aversion often lead to much longer development cycles for production systems. ### Quality Assurance and Validation The production system includes comprehensive quality assurance processes that are essential for pharmaceutical applications: **Business Domain Validation**: Subject matter experts from various R&D domains actively validate the tool's outputs. These domain experts assess whether the AI-generated insights align with their professional expertise and experience. **Sprint-Based Testing**: The team conducts rigorous testing processes every sprint, ensuring continuous quality improvement and stability. **Version Benchmarking**: New versions are systematically benchmarked against previous versions to ensure that updates improve rather than degrade performance. **Answer Correctness Verification**: Given the critical nature of pharmaceutical research, the team places heavy emphasis on ensuring that AI-generated answers are factually correct and professionally sound. ### Guardrails and Safety Measures The production system incorporates multiple layers of guardrails appropriate for pharmaceutical R&D applications: - **Human-in-the-loop validation** by domain experts - **Transparent reasoning** showing how conclusions were reached - **Source traceability** allowing users to verify information against original data - **Version control** and rollback capabilities for system updates - **Comprehensive testing** before any production changes ## User Adoption and Change Management ### Addressing Change Fatigue AstraZeneca recognized that change fatigue is a real phenomenon affecting AI adoption across the organization. Senior leadership actively supports AI initiatives, but the company invested heavily in structured change management practices to ensure successful adoption. The change management approach includes several components: - **Cross-functional collaboration** involving HR, legal, and business groups - **AI accelerator teams** providing guidance and best practices - **Consistent narrative** across the organization about AI goals and benefits - **Regular showcases and spotlights** to demonstrate AI capabilities and successes ### AI Accreditation Program A particularly notable aspect of AstraZeneca's LLMOps implementation is their comprehensive AI accreditation program. This top-down initiative includes multiple tiers of certification and curriculum designed to help employees understand how AI will change their work and how to effectively adopt and utilize AI tools. The program features: - **Four-tier certification structure** with progressive skill development - **Rewards and recognition** through certificates and LinkedIn sharing - **Senior leadership participation** with AI learning as part of individual goals - **Lifelong learning encouragement** with recommendations for daily upskilling ### Business Integration Strategy Rather than implementing AI as a standalone technical solution, AstraZeneca enlisted product managers from different business areas to integrate the Development Assistant into actual workflows and daily tasks. This approach ensures that the AI system provides real business value rather than just technical novelty. The integration strategy involves: - **Starting small** with simple use cases and gradually expanding - **Cross-functional teams** bringing together business, IT, and data expertise - **Workflow integration** rather than separate AI applications - **Continuous feedback loops** from actual users in production scenarios ## Lessons Learned and Technical Insights ### Collaboration with Cloud Providers AstraZeneca's partnership with AWS was critical to their success, involving hands-on collaboration with AWS technical teams throughout the development and deployment process. This partnership provided access to specialized expertise in multi-agent architectures and production AI deployment patterns. ### Multi-Agent Architecture Benefits The evolution from single-agent to multi-agent architecture provides valuable insights for other organizations: - **Scalability**: Multi-agent systems can handle larger data volumes and more complex queries - **Specialization**: Domain-specific agents can provide more accurate and relevant responses - **Flexibility**: New domains can be added by introducing new agents rather than retraining the entire system - **Performance**: Distributed processing across specialized agents improves overall system responsiveness ### Data Quality Importance The case study reinforces the critical importance of data quality and metadata in LLMOps implementations. Poor metadata and inconsistent vocabulary can significantly impact LLM performance, while well-structured data products with controlled vocabularies enable more accurate and reliable AI systems. ### Production-First Mindset AstraZeneca's focus on reaching production rather than remaining in proof-of-concept mode represents a mature approach to AI implementation. Their willingness to move beyond experimentation to operational deployment demonstrates the organizational commitment necessary for successful LLMOps. ## Results and Impact While specific quantitative metrics were not provided in the presentation, AstraZeneca reports several positive outcomes: - **Time savings**: Tasks that previously took hours can now be completed much more quickly - **User validation**: Domain experts confirm that the tool provides valuable insights that align with their professional judgment - **Scaling success**: The system has successfully expanded beyond its initial clinical trial focus to other R&D domains - **User growth**: The platform is actively expanding its user base while maintaining quality and reliability The Development Assistant represents a successful example of enterprise LLMOps implementation that balances technical sophistication with practical business needs, demonstrating how pharmaceutical companies can leverage AI to accelerate drug development while maintaining the rigorous quality standards required in healthcare applications.