A large public healthcare company specializing in radiology software deployed an AI-powered automation solution to streamline the complex process of procedure code selection during patient appointment scheduling. The traditional manual process took 12-15 minutes per call, requiring operators to navigate complex UIs and select from hundreds of procedure codes that varied by clinic, regulations, and patient circumstances. Using PromptQL's domain-specific LLM platform, non-technical healthcare administrators can now write automation logic in natural language that gets converted into executable code, reducing call times and potentially delivering $50-100 million in business impact through increased efficiency and reduced training costs.
This case study presents a compelling example of LLMOps implementation in healthcare through Hasura/PromptQL’s work with a large public healthcare company that creates software for radiologists and radiology clinics. The partnership demonstrates how domain-specific LLM platforms can address complex business automation challenges that traditional rule-based systems struggle to handle effectively.
The healthcare company faced a significant operational bottleneck in their patient appointment scheduling process. When patients called to schedule appointments, operators needed 12-15 minutes per call to navigate through complex enterprise software interfaces, gather patient information, and most critically, determine the correct medical procedure codes. The presenter noted that reducing call times by just 3 minutes could yield approximately $50 million in business impact across their network servicing thousands of clinics globally, primarily in the US and Europe.
The core technical challenge centered around what the presenter calls “the automation paradox” - the people who understand the business rules (healthcare administrators) cannot code the automation, while the people who can code (developers) don’t understand the domain-specific rules. This creates a fundamental bottleneck in traditional software development approaches.
The procedure code selection process exemplifies this complexity. Different medical scenarios require different codes - for instance, a mammogram might have codes like “mm123” for basic screening, but become “mm123wa” if wheelchair assistance is needed. The permutations become explosive when considering factors such as:
The existing solution involved developers creating extensive configuration systems to handle edge cases, leading to configuration explosion, increased training burden for operators, and significant maintenance overhead. Many business rules remained uncoded because the cost of encoding them exceeded their perceived benefit.
PromptQL’s solution addresses this challenge through a multi-layered LLM architecture that bridges the gap between natural language business requirements and executable code. The system introduces several key components:
Domain-Specific Language Generation: Instead of having developers use foundation models with general programming knowledge, the platform creates company-specific query languages (referred to as “AcmeQL” in the presentation). Non-technical users interact with models that have been trained on the specific domain language and ontologies of their organization.
Natural Language to Executable Logic Pipeline: The system converts natural language business rules into deterministic, executable plans written in the company-specific query language. This creates a crucial abstraction layer that maintains the precision needed for production systems while enabling natural language interaction.
Semantic Layer Integration: The platform requires extensive setup of semantic layers that encode domain-specific terminology, entities, procedures, and ontologies. This ensures that when business users reference domain concepts, the LLM understands the specific context and constraints of their organization.
The case study demonstrates several important LLMOps practices through a GitHub issue assignment demo that parallels the healthcare use case. The system supports an iterative development process where non-technical users can:
The testing approach is particularly noteworthy. When the user discovered that “Tom” was being assigned issues but belonged to an external company, they could immediately add exclusion rules in natural language: “remove somebody from an external company.” The system then re-tests the automation against various scenarios to ensure the new rule works correctly across different inputs.
Security and Authorization Model: The platform addresses security concerns through a multi-tenant architecture where the domain-specific query language (AcmeQL) runs strictly in user space rather than data space. This containment approach allows extensive “vibe coding” by non-technical users while maintaining security boundaries and preventing unauthorized data access.
A significant innovation in this LLMOps implementation is the creation of software development lifecycle (SDLC) processes designed specifically for non-technical users. The platform abstracts away traditional DevOps complexity while maintaining essential practices like:
This represents a fundamental shift in how LLMOps platforms can democratize software development while maintaining production-quality standards.
The healthcare implementation reportedly delivers substantial business value, with the presenter claiming $100 million or more in impact that the company expects to realize over the course of the year. This impact stems from multiple sources:
The case study reveals several important architectural decisions that enable successful LLMOps deployment:
Model Specialization: Rather than using general-purpose foundation models, the platform invests heavily in domain-specific model training and fine-tuning. This specialization is crucial for handling the nuanced terminology and business rules specific to healthcare operations.
Deterministic Execution: The intermediate query language ensures that despite the non-deterministic nature of LLM interactions, the final execution is completely deterministic and auditable. This is particularly important in healthcare where compliance and traceability are essential.
Data Layer Separation: The architecture maintains clear separation between the AI-powered business logic layer and the underlying data access layer, ensuring that security and authorization rules are enforced consistently regardless of how business logic is authored.
While the case study presents impressive results, several challenges and limitations should be considered:
Domain Complexity: The system requires extensive upfront investment in creating domain-specific semantic layers and ontologies. This setup cost may be prohibitive for smaller organizations or less complex use cases.
Model Training and Maintenance: Maintaining domain-specific models requires ongoing investment in training data curation, model updates, and performance monitoring. The case study doesn’t detail these ongoing operational requirements.
User Adoption: Successfully transitioning non-technical users from traditional interfaces to conversational AI systems requires change management and training, though presumably less than traditional programming approaches.
Validation and Testing: While the demo shows testing capabilities, the case study doesn’t deeply address how complex business rules are validated for correctness, especially when they interact with each other in unexpected ways.
This case study demonstrates the potential for LLMOps to address a fundamental challenge in enterprise software: the gap between business domain expertise and technical implementation capability. The healthcare industry, with its complex regulations, varying operational requirements, and high stakes for accuracy, provides an excellent test case for these approaches.
The success of this implementation suggests that similar approaches could be valuable across other heavily regulated industries where business rules are complex, frequently changing, and require domain expertise to implement correctly. Industries like finance, insurance, and legal services could benefit from similar domain-specific LLM platforms.
The case study also highlights the importance of building LLMOps platforms that are specifically designed for non-technical users, rather than simply providing better tools for developers. This represents a significant shift in how organizations might structure their technology teams and development processes in the age of AI.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Anthropic developed a production multi-agent system for their Claude Research feature that uses multiple specialized AI agents working in parallel to conduct complex research tasks across web and enterprise sources. The system employs an orchestrator-worker architecture where a lead agent coordinates and delegates to specialized subagents that operate simultaneously, achieving 90.2% performance improvement over single-agent systems on internal evaluations. The implementation required sophisticated prompt engineering, robust evaluation frameworks, and careful production engineering to handle the stateful, non-deterministic nature of multi-agent interactions at scale.
Rippling, an enterprise platform providing HR, payroll, IT, and finance solutions, has evolved its AI strategy from simple content summarization to building complex production agents that assist administrators and employees across their entire platform. Led by Anker, their head of AI, the company has developed agents that handle payroll troubleshooting, sales briefing automation, interview transcript summarization, and talent performance calibration. They've transitioned from deterministic workflow-based approaches to more flexible deep agent paradigms, leveraging LangChain and LangSmith for development and tracing. The company maintains a dual focus: embedding AI capabilities within their product for customers running businesses on their platform, and deploying AI internally to increase productivity across all teams. Early results show promise in handling complex, context-dependent queries that traditional rule-based systems couldn't address.
