## Overview This case study from John Snow Labs presents an ambitious end-to-end generative AI solution for healthcare data integration. Rather than focusing on fine-tuning a single model, the approach rethinks how healthcare systems can ingest, normalize, and query patient data across multiple modalities. The goal is to take raw, messy, and fragmented healthcare data and transform it into unified patient journeys that can be queried using natural language. The presenter emphasizes that patient journeys in healthcare are inherently complex and messy. A single cancer patient in the US generates over 1,000 pages of text annually, with significant duplication, copy-pasted data, and occasional errors. Patients see multiple doctors, move between hospitals, and have records scattered across different systems. The solution aims to handle this complexity while meeting enterprise production requirements. ## Problem Statement Healthcare organizations face several critical challenges when trying to understand their patient populations: The data exists in multiple formats and modalities, including structured EHR data (from systems like Epic Clarity), unstructured clinical notes (discharge summaries, radiology reports), semi-structured data (FHIR resources, HL7 messages), PDFs, CSV files, and claims data. This data is not normalized to common terminologies, may be inconsistent or contradictory, and continuously updates over time. Traditional approaches require extensive data pre-processing, manual coding, and specialized teams to answer even relatively simple analytical questions. The presenter notes that translating a question like "give me patients diagnosed with back pain who had spinal fusion" into executable SQL is surprisingly difficult because these clinical concepts map to numerous specific diagnosis and procedure codes. ## Architecture and Technical Approach The solution implements a multi-stage pipeline with several healthcare-specific LLMs working in concert: ### Information Extraction The first stage reads unstructured data and extracts entities in a completely automated way. Critically, the system also understands the context of those entities. The presenter gives examples showing that "this is a woman with breast cancer" versus "this is a woman without breast cancer" versus "her mother had breast cancer at 55" are completely different clinical assertions even though they contain the same terms. The system must also extract semantic relationships including temporal relationships, anatomical locations (right lung vs. left lung, which kidney), and medication dosages. ### Terminology Mapping Extracted entities must be mapped to standard clinical codes. This is described as a traditionally labor-intensive process with an entire industry built around terminology services. The goal here is to automate this mapping, taking free-text mentions and mapping them to standardized codes that enable consistent querying. ### Temporal Normalization Understanding dates and relative time references is essential for building accurate patient timelines. The example given: if a visit summary from June 1st says "two weeks ago he started feeling knee pain," and someone queries in September for "patients with knee pain in the last six months," the system must normalize "two weeks ago" to mid-May to correctly answer the query. ### Deduplication and Merging This is described as one of the most interesting reasoning problems in the system. When the same clinical fact appears multiple times with different specificity or conflicting values, the system must apply domain-specific logic. Examples include: - If one entry says "pain" and another says "knee pain" on the same day, prefer the more specific "knee pain" - For three weight measurements on the same day (50kg, 51kg, 52kg), take the average - If a medication prescription differs from pharmacy claims, prefer what the pharmacy actually dispensed (e.g., the generic substitution) Each clinical effect type requires its own reasoning rules for handling conflicts and uncertainties. ### Data Model Choice The system outputs to the OMOP Common Data Model, a relational database standard. This choice was deliberate despite knowledge graphs being a more natural logical representation. Two reasons are cited: First, OMOP is an industry standard with a rich ecosystem of open-source tools for risk scoring, data quality, and code creation. Second, at scale (50 million patients, 3-4 billion documents), relational databases are far easier to operate in production. The team knows how to do indexing, views, sharding, backup, and patching for relational databases, whereas newer graph databases present operational challenges. ## Production Requirements and Enterprise Constraints The presenter emphasizes that the system must work "for real" in production healthcare environments. This translates to several concrete requirements: **PHI Compliance and Security**: The system must be designed to process protected health information (PHI), must not share data externally, and must be capable of running in air-gapped environments without calling external APIs. **Accuracy and Explainability**: The system uses healthcare-specific LLMs, can explain its answers, and provides citations to source data. When a doctor sees an answer that doesn't seem right, they can trace back to the original source. **Consistency**: Asking the same question twice must return the same answer. The presenter notes this is a critical requirement for clinical trust—if the system gives different patient lists for the same query on repeated calls, clinicians will lose trust immediately. **Operational Simplicity**: The system runs on commodity infrastructure that DevOps and SecOps teams understand. No proprietary data models or exotic infrastructure that requires specialized knowledge. **Scale**: Must support millions of patients (10+ million mentioned) with billions of documents and inputs. **Interoperability**: The structured output should be accessible not just through a chatbot but also through dashboards, notebooks, and as input to other machine learning models. ## Natural Language Querying and Text-to-SQL The final layer enables natural language querying over the unified patient data. Users can ask questions like "give me all patients with stage three cancer that are triple negative and have not yet started immunotherapy" or "patients who have had this side effect in the last six months." The presenter specifically addresses why they couldn't use general-purpose LLMs like GPT-4 for this text-to-SQL task. Three critical issues were identified: **Accuracy**: When shown an example query, GPT-4 generated a complex SQL statement that looked reasonable but was "just plain incorrect" and returned wrong results. The SQL queries required for healthcare are far more complex than standard text-to-SQL benchmarks, so even models that perform well on benchmarks fail here. **Completeness**: The GPT-4 output included commented lines saying "please replace this with the correct code" rather than providing complete, executable SQL. **Consistency**: General-purpose models don't guarantee consistent outputs. Asking the same question multiple times might return different patient sets, which destroys clinical trust. **Performance**: Even if the SQL were correct, it must be optimized for the specific database schema with proper use of indices and materialized views. At billion-fact scale, users cannot accept a 5-second response for one query and a 5-minute response for a slight variant. This led to the decision to fine-tune healthcare-specific LLMs for natural language querying that understand the OMOP data model and can generate optimized, consistent, accurate SQL. ## Demonstrated Value of Multi-Modal Integration The case study demonstrates the value of integrating multiple data sources through a detailed patient example. For one patient tracked over 10+ years: With only structured EHR data: Three hospitalizations were visible across 2015, 2017, and 2019. Adding unstructured data (discharge notes, radiology reports): Four additional years of data became visible (going back to 2011), including mammography and preventive care that was never captured in structured fields. Adding FHIR resources: A previously unknown hospitalization was discovered—the patient had been treated at a different hospital while traveling out of state, and this data only came through a health information exchange via FHIR. The presenter cites published studies showing that over half of entered diagnosis codes were inappropriate and about a quarter were omitted, while nearly 40% of important diagnoses were only mentioned in free text. Critical information like family history, medications taken at home, and treatment response often exists only in unstructured notes. ## Multiple LLM Components The solution is described as a "multi-LLM solution" with different models handling different tasks: - Information extraction from unstructured text - Semantic information extraction (understanding context and relationships) - Merging and deduplication reasoning - Natural language question answering - Clinical inference (risk scores, patient conditions) This reflects a sophisticated LLMOps architecture where different specialized models are orchestrated to handle different aspects of the pipeline, rather than trying to use a single general-purpose model for everything. ## Honest Assessment While the presentation makes compelling claims, a few caveats are worth noting. The system appears to be a commercial product from John Snow Labs, so the presentation naturally emphasizes benefits. The specific accuracy metrics, latency figures, and scale benchmarks beyond "millions of patients" are not provided in detail. The comparison to GPT-4 for text-to-SQL is illustrative but represents one example rather than a systematic benchmark. That said, the architectural decisions (OMOP for operability, fine-tuned domain models for accuracy, relational databases for scale) reflect practical production experience, and the specific examples of multi-modal data value are grounded in cited research.