Hinge Health faced the challenge of deploying AI systems directly to healthcare members in an environment where consumer health AI has demonstrated serious safety failures, including misdiagnoses and dangerous medical advice. Their solution centered on a three-layer architectural approach: stripping PHI at pipeline boundaries rather than at runtime, implementing deterministic code layers above the model for irreversible decisions like emergency escalation, and establishing continuous evaluation through automated judges, member feedback, and manual trace sampling. The approach emphasizes that safety failures are primarily architectural decisions rather than model failures, and that launch represents the beginning of risk management rather than its conclusion. The system combines architectural constraints with human decision-making frameworks to maintain member trust while scaling healthcare AI capabilities.
Rashi Agarwal, who leads AI and ML at Hinge Health, presented a detailed practitioner’s view of deploying member-facing healthcare AI systems in production. The presentation addresses the significant safety challenges facing healthcare AI in 2026, where despite 40 million people using frontier models for healthcare triage, serious safety incidents have occurred. These include a case where an AI chatbot recommended bromine sodium bromide as a salt substitute, leading to a hospitalization with bromide levels 200 times the safe limit, and independent safety testing from Mount Sinai showing that consumer health AI under-triages life-threatening emergencies 50% of the time. ECRI, a patient safety organization, named AI chatbot misuse as the number one health technology hazard of 2026. The presentation operates from the premise that these are not frontier model problems but rather represent the production baseline that healthcare organizations must address through careful architectural and operational decisions.
The approach at Hinge Health rests on three fundamental principles that shape their entire production LLM strategy. First, the constraint is the architecture - most AI safety failures in healthcare are not model failures but architectural decisions made before any tokens are generated. Second, deterministic rules belong above the model, not inside it, because what can never be wrong cannot be left to probability. Third, safety is a continuous evaluation layer rather than a one-time gate, with the product launch representing where real risk starts rather than where it ends. These principles structure both the technical implementation and the organizational decision-making around healthcare AI deployment.
The first major layer of the Hinge Health production system focuses on protecting Protected Health Information through architectural rather than policy-based approaches. While policy defines what to protect, architecture ensures protection actually happens. The fundamental shift in their approach is treating PHI as a pipeline boundary problem rather than a runtime problem. Rather than redacting PHI when logs are written to dashboards, their architecture strips PHI at ingestion, before data ever reaches the data lake. By the time data is stored, PHI is already removed, meaning developers opening dashboards have nothing to redact because PHI was never there in the first place.
The architecture maintains complete separation between production and non-production environments with no pipes between them, recognizing that even a single connection point could allow member data to leak into development environments. Given that HIPAA laws are extremely stringent in healthcare, with regulatory bars much higher than in other industries, this architectural separation is non-negotiable. Access control depends on two factors: role and geographic region. Because teams are geographically distributed but not everyone has access to PHI, certification is a policy applied only to specific regions. Engineers outside regulated regions cannot reach raw PHI at all, regardless of their role.
Critically, compliance rules including HIPAA, FDA’s good machine learning practice, and state laws like Texas triaging regulations are not afterthoughts but rather grounding inputs for system design. The approach explicitly rejects attempting to layer HIPAA compliance on top of an existing architecture, instead starting with compliance requirements and allowing architecture to grow around them. When PHI is protected at the architectural level, the system becomes incapable of certain failure modes rather than merely relying on policies being followed.
The second layer addresses the fundamental limitation that probabilistic systems excel at generation but are unreliable for behaviors that can never be wrong. The guiding rule is simple: must-not-fail behavior belongs above the prompt and above the model. This means implementing a code layer that runs first on every conversational turn before the model executes. This code layer makes irreversible decisions including whether an emergency escalation is needed, whether routing to 911 is required, and whether a clinician must step into the loop. The model handles the long tail of conversations and interactions, but never gets a vote on high-stakes calls.
The conceptual model is a stack with code on top and the model below. Every turn passes through the code layer first. While most turns do reach the model, the model is excluded from high-stakes decisions. This reflects a fundamental principle: a model is not a guardrail, and a model with a system prompt is also not a guardrail. Code that runs above the model approaches true guardrails. Even the labs building frontier models publish authority hierarchies showing root, system, developer, user, and guideline layers, where every layer above user is one prompt injection away from being overridden. If the labs themselves don’t trust the prompt as a security boundary, production healthcare systems shouldn’t either.
The code layer implements several critical functions. For emergency escalation, if a member mentions self-harm, suicidal ideation, or acute medical emergencies, the system must route to 911 or 988 immediately, with the model never seeing the turn. Code runs first, decides, and routes. For intent routing in the multi-agent architecture, code determines which capability handles a conversation turn: clinical, tech support, education from accredited articles, or exercise recommendation. While the model can help classify, high-stakes routing paths must take deterministic routes at the top level, preventing clinical questions from being quietly routed to generic tech support agents, which would be unrecoverable. For identity verification, anything touching member data must check that the right member is on the other end through authentication checks, which represent security boundaries rather than prompt-based controls. The underlying pattern across all three cases is that code runs first, makes irreversible decisions, and leaves everything else to the model.
The third layer implements safety as a continuous evaluation layer running throughout the system’s operation rather than as a pre-launch checklist. While most teams treat evaluations as pre-launch gates that they run once before shipping, Hinge Health recognizes this is necessary but insufficient. What actually holds up in production are judges that continuously score real conversations as they happen, evaluating live traffic rather than saved golden datasets. These signals come from three distinct sources, each catching different failure modes.
First, automated judges score conversations across 30 to 40 or more dimensions that continuously refresh. These dimensions include clinical accuracy, safety, escalation appropriateness, relevance, drift detection, refusal behavior, and many others. Automated judges catch regressions and even sensitive drops in quality as they occur. Second, member feedback through thumbs up and thumbs down on individual messages provides the truth signal coming directly from the people being served. This feedback catches tone problems and issues that automated judges miss. Third, sample traces involve random samples spread across capabilities, with high-stakes cases receiving 100% sampling where humans read every single interaction. Ultimately, people must read these signals because humans catch what no single metric can capture.
The critical bottleneck identified is not compute, models, or capability, but rather having enough people to read signals and act on them. Some failures cannot be prompted away - they’re shipped as fixes, return under new conditions with new prompts and tools or when models shift, get fixed again, and each round buys progressively less improvement with failure rates never hitting zero. At this point, monitoring becomes not a last resort but the first resort that is always on. Each new failure observed in production generates a new judge, requiring the underlying architecture and system to scale with new judges and monitoring as the consumer base grows. Monitoring is ultimately how the team knows whether their architecture is still holding.
The second major section of the presentation addresses what happens when architecture alone cannot resolve issues and humans must make decisions about what ships versus what is held back. A typical scenario involves being five days from launching a new capability when one issue remains on the board, with five different stakeholders seeing five different risks. Clinical teams see member safety risk and want to hold the launch. Legal sees regulatory exposure. Compliance sees audit risk. Product sees adoption risk where shipping a broken feature means it won’t land with users. Engineering sees velocity risk where fixing the issue would slip the date. Five rational people see five different risks with five different proposed fixes, raising the question of whether to hold and fix or to ship.
The framework for making these decisions consists of five rules. Rule one is that worst case always wins - severity is set by the worst plausible outcome rather than the average. A bug that lightly annoys 100% of users is far less severe than one that could cause serious harm in 0.1% of cases. This is non-negotiable in healthcare. When triaging, the question is not “How often does this happen?” but rather “What’s the worst version of this?” which sets the severity level.
Rule two states that severity is not capacity, keeping politics out of decisions. A bug’s severity derives from the harm it causes, not from who owns it, whether the team has capacity to fix it, or how hard the fix would be. Three options exist: fix, delay the launch, or accept the risk with explicit sign-off. The team never quietly downgrades a bug simply because they cannot address it immediately.
Rule three implements asymmetric defaults - when uncertain, always pick the safer mistake. For safety bugs, the mathematics is one-sided: shipping a real safety bug is much worse than delaying for a false alarm, so when uncertain, always hold and fix. For polish bugs, the math runs the opposite direction, where delaying a launch costs more than shipping a small flaw, so when uncertain, ship. The framework doesn’t make decisions but rather indicates which direction to lean.
Rule four focuses on revealed risk tolerance rather than stated risk tolerance. The launch bar is what the organization already accepts in production, not what it claims it will accept. If a behavior has been live in the existing product for weeks or months without escalation, member complaints, or leadership concern, it cannot be called a launch blocker for new features. While stated risk tolerance might be “no bugs in production,” revealed risk tolerance reflects what is actually shipping today, and teams should calibrate to this revealed tolerance as the floor.
Rule five acknowledges that humans are the constraint. While judges scale and code traces automatically, with dashboards refreshing every few hours, the hard part is having enough people to read signals and act on them. Additionally, fast follows are committed debt rather than optional backlog items - if something didn’t ship at launch, it’s already committed rather than being a wish list item.
An important operational discipline emerges from working with non-deterministic systems: in such systems, judges are also non-deterministic, requiring verification of the scorer before trusting the score. A concrete example illustrates this principle. Suppose a clinical accuracy judge has maintained a steady score of 4.9 for weeks, then drops to 4.5 and stays there. The immediate instinct might be to start changing prompts, assuming the agent is broken. This reactive approach is risky because it fixes one thing while potentially breaking another, and worse, it changes the agent based on a signal that might not be true.
The proper discipline requires first asking whether the judge is right. Consider a scenario where a member asks about caffeine and the agent provides FDA standard guidance stating 400 mg for most adults, less if pregnant or on certain medications. If the judge flags this as a hallucination because the agent mentioned pregnancy and medications without checking member status, the judge is actually over-calling - the agent simply provided clinical context. In this case, fix the judge. In contrast, if the agent says 1,000 mg a day is fine, well above safety limits, and the judge correctly flags it, the agent is wrong and should be fixed. The rule is always to ask whether the judge is right before changing the agent’s response. Fixing a judge prompt is not cheating because judges are software too and must continuously evolve. This represents what production discipline looks like when systems are non-deterministic.
The entire presentation distills into six takeaways, three from architecture and three from decisioning. On the architecture side, the pattern follows “Don’t X what you can Y”: don’t policy what you can architect, don’t prompt what you can code, and don’t gate what you can monitor. On the decisioning side, the pattern addresses how humans decide when systems cannot: score by the worst case and default to the safer mistake, calibrate to organizational revealed preferences, and always design for human in the loop, treating fast follows as debt rather than backlog.
The presentation acknowledges that building guardrails first is slower than bolting them on later, but frames this as design rather than limitation. The goal is not building a generic low-stakes chatbot but rather building a system worthy of someone’s health. The architecture defines how to build it, decisioning frameworks define when to ship, and member trust defines why it matters.
The presentation provides valuable insights into production healthcare AI from a true practitioner perspective, with the specificity of implementation details lending credibility. The three-layer architectural approach reflects sophisticated thinking about where different types of controls belong in the stack. The emphasis on architectural prevention over runtime detection represents sound engineering for high-stakes applications.
However, several aspects warrant balanced consideration. The presentation heavily emphasizes deterministic controls and architectural constraints, which is appropriate for healthcare, but may represent overcorrection from the prevalent pattern of treating prompts as security boundaries. The specific split between deterministic routing and model-based generation likely requires continuous calibration as models improve. The framework also doesn’t deeply address how to determine which decisions truly must be deterministic versus which could be safely model-based with sufficient monitoring.
The continuous evaluation approach with 30-40 automated judges plus human sampling represents significant operational overhead. While the presentation acknowledges that humans are the constraint, it doesn’t fully address the scalability economics of this approach or how smaller healthcare organizations might implement similar safety standards with more limited resources. The revealed risk tolerance concept is pragmatic but could potentially be used to justify lowering standards rather than maintaining them.
The decision framework’s five rules provide clear guidance but assume certain organizational maturity levels. The principle that “worst case always wins” is appropriate for healthcare but requires sophisticated capability to estimate worst-case scenarios accurately, and the asymmetric defaults for safety versus polish bugs require clear categorization that may not always be obvious. The presentation would benefit from more discussion of edge cases where these frameworks produce ambiguous guidance.
Overall, the approach represents mature thinking about production LLM deployment in high-stakes environments, with the architectural emphasis on fail-safe design and continuous monitoring reflecting lessons learned from actual production operations. The integration of technical architecture with human decision-making frameworks acknowledges the sociotechnical reality of deploying AI systems. The candid discussion of industry safety failures and the acknowledgment of constraints like human bandwidth provide realistic grounding often missing from vendor presentations.
Benchling, a 14-year-old platform for life sciences R&D data management, launched Benchling AI six months ago to bring intelligent agents to scientific workflows. The problem scientists face is the time-consuming nature of drug discovery, from initial experiments to FDA submissions, involving manual data entry, analysis, and report writing. Benchling AI addresses this through a chat-based agent interface that leverages their extensive historical data repository to help scientists find relevant experiments, design new tests, analyze results, and generate regulatory reports. The system uses multiple model families in parallel for critical tasks like data entry, employs custom-built harnesses tailored to scientific workflows rather than coding-focused architectures, and integrates agent skills that function like standard operating procedures. Early results suggest the potential to reduce drug discovery timelines by 2x through eliminating workflow bottlenecks and enabling more efficient experimental design.
Medable developed Agent Studio, a comprehensive platform for deploying AI agents in clinical trial operations to address the lengthy drug approval process that currently takes over 10 years. The platform enables both internal teams and customers to build configurable multi-agent systems that tackle problems like document classification in electronic trial master files and clinical research monitoring across multiple data systems. By taking a platform-first approach with support for model-agnostic agents, RAG knowledge integration, MCP connectors, workflow functionality, and robust evaluation frameworks, Medable has deployed multiple agentic applications that help clinical research associates process over 80,000 documents per year and monitor data across 13+ disparate systems, with the ambitious goal of reducing clinical trial timelines from 10 years to one year.
Abridge, a clinical intelligence platform, developed an AI system to transform unstructured patient-clinician conversations into clinical documentation, addressing the significant burden of documentation that causes physician burnout. The company processes over 100 million conversations annually across 250 major U.S. health systems, using LLM-based tools to generate clinical notes and provide agentic assistance to clinicians. Through significant investment in evaluation infrastructure, including migration to LangGraph and LangSmith, implementation of both reference-free and reference-based judges, and rigorous testing processes including AB testing in healthcare settings, Abridge reduced their release cycle from 1-2 months to just days while maintaining the extremely high quality and safety standards required in healthcare.
