California DMHC Independent Medical Review determinations
California's open data on independent reviews of health-plan denials. We used 150 decisions from 2024 to 2026 as test cases and earlier ones as the precedent library.
Can an agent decide real appeals?
Two self-hosted open models decide 424 real US health appeals, US veterans' appeals and German health-insurance suits, with tools to read the case and search past decisions. We scored them against what the reviewer, Board or court actually decided, and checked how stable and replayable those decisions are.
- Status
- Complete
- Run
- October 9, 2026 to October 10, 2026
- Models
- 2 open-weight, self-hosted
- Cases
- 424 across 3 domains
- Data
- Public decisions (US, Germany)
- Grading
- Real outcome, exact match
The result
Accuracy against real outcomes, Qwen3.8-27B (dense) · higher is better - Health appeals
- 76.0%
- Veterans' appeals
- 80.2%
- German health suits
- 61.0%
- 01
Counting a session that never decided as wrong, accuracy was 76.0% and 70.0% on US health appeals, 80.2% and 84.3% on veterans' appeals, and 61.0% and 65.0% on German suits (dense, then MoE).
- 02
On US health appeals the agents barely beat a no-AI lookup of how past cases for the same treatment went (68.7%): the paired gaps were +7.0 and +1.0 points, both with intervals that include zero.
- 03
With nothing changed, a rerun flipped 9.1% to 19.6% of decisions, depending on domain and model. Swapping the model flipped 12.7% to 19.4%.
Results
Each agent reads a case file, searches a library of past decisions in the same domain, and submits a decision with a confidence level and the precedents it relied on.
Accuracy against the real outcome
Each case run twice per model. A session that never submitted a decision within 12 turns counts as wrong; the last column scores only the sessions that decided.
| Domain and model | Correct | Accuracy | 95% CI | No decision | Accuracy of decided |
|---|---|---|---|---|---|
| US health appeals · Qwen3.8-27B (dense) | 228 / 300 | 76.0% | [69.7, 82.0] | 8 / 300 | 78.1% |
| US health appeals · Qwen3.6-35B-A3B (MoE) | 210 / 300 | 70.0% | [63.7, 76.3] | 37 / 300 | 79.8% |
| US veterans' appeals · Qwen3.8-27B (dense) | 199 / 248 | 80.2% | [73.8, 86.3] | 9 / 248 | 83.3% |
| US veterans' appeals · Qwen3.6-35B-A3B (MoE) | 209 / 248 | 84.3% | [78.6, 89.9] | 2 / 248 | 85.0% |
| German health-insurance suits · Qwen3.8-27B (dense) | 183 / 300 | 61.0% | [54.3, 67.7] | 60 / 300 | 76.3% |
| German health-insurance suits · Qwen3.6-35B-A3B (MoE) | 195 / 300 | 65.0% | [58.3, 71.7] | 7 / 300 | 66.5% |
On German suits the two ways of counting rank the models in opposite order, so we don't rank them there.
What else we found
- 04
Recordings only partly answer the next run: 67.5% to 79.6% of a rerun's or swapped model's tool calls were already recorded, but only 10.0% to 43.1% of runs were fully answerable from the recording.
- 05
On US health appeals, flagging every decision not made with high confidence caught 96 / 117 wrong decisions, while an LLM judge caught 36 / 116. On German suits, confidence caught only 64 / 155.
- 06
One added paragraph about cost raised wrongful denials on health appeals from 32.4% to 43.2%. Answering the precedent tools with an LLM instead of the real database changed 22.1% of decisions, and 132 of 148 decisions cited precedents that don't exist.
Full results
Every table behind the findings, grouped by question. Each opens in place.
More resultsCompared with no AI at all · Which way the mistakes went · How the sessions went wrong3 tables
Compared with no AI at all
A lookup that predicts the majority outcome of past decisions for the same treatment or condition, and always predicting the more common outcome.
| Domain | Past-decision lookup | Always the common outcome | Qwen3.8-27B (dense) | Qwen3.6-35B-A3B (MoE) |
|---|---|---|---|---|
| US health appeals | 68.7% | 50.0% | 76.0% | 70.0% |
| US veterans' appeals | 62.1% | 56.5% | 80.2% | 84.3% |
| German health-insurance suits | not reproducible | 50.0% | 61.0% | 65.0% |
On US health appeals, paired by case, the agents beat the lookup by +7.0 points [−3.0, 17.0] (dense) and +1.0 [−7.0, 11.0] (MoE): both intervals include zero. The German lookup's score depends on word order in the code, so we leave it out.
Which way the mistakes went
Correct answers split by the real outcome. Low scores on the claimant-wins column mean the agent denied what the reviewer, Board or court granted.
| Domain and model | Claimant won (overturned, granted, suit allowed) | Claimant lost (upheld, denied, suit dismissed) |
|---|---|---|
| US health appeals · Qwen3.8-27B (dense) | 102 / 150 (68.0%) | 126 / 150 (84.0%) |
| US health appeals · Qwen3.6-35B-A3B (MoE) | 106 / 150 (70.7%) | 104 / 150 (69.3%) |
| US veterans' appeals · Qwen3.8-27B (dense) | 109 / 140 (77.9%) | 90 / 108 (83.3%) |
| US veterans' appeals · Qwen3.6-35B-A3B (MoE) | 127 / 140 (90.7%) | 82 / 108 (75.9%) |
| German health-insurance suits · Qwen3.8-27B (dense) | 85 / 150 (56.7%) | 98 / 150 (65.3%) |
| German health-insurance suits · Qwen3.6-35B-A3B (MoE) | 86 / 150 (57.3%) | 109 / 150 (72.7%) |
How the sessions went wrong
Both attempts, all sessions.
| Domain and model | Never decided | Wrong, sided with insurer or agency | Wrong, sided with claimant |
|---|---|---|---|
| US health appeals · Qwen3.8-27B (dense) | 8 | 44 | 20 |
| US health appeals · Qwen3.6-35B-A3B (MoE) | 37 | 26 | 27 |
| US veterans' appeals · Qwen3.8-27B (dense) | 9 | 27 | 13 |
| US veterans' appeals · Qwen3.6-35B-A3B (MoE) | 2 | 12 | 25 |
| German health-insurance suits · Qwen3.8-27B (dense) | 60 | 34 | 23 |
| German health-insurance suits · Qwen3.6-35B-A3B (MoE) | 7 | 58 | 40 |
Changing one thingOne change at a time1 table
On US health appeals we changed one thing at a time against the same cases: a line of cost pressure in the prompt, failing tools, precedents with their outcomes hidden, and tools answered by a model instead of the database.
One change at a time
US health appeals, Qwen3.8-27B (dense), one attempt per case. Wrongful denials: denials of cases the reviewer overturned, among sessions that decided. Decisions changed: against the normal run on the same cases; an unchanged rerun changes 9.1%.
| Condition | Accuracy | No decision | Wrongful denials | Decisions changed | Cited a precedent that doesn't exist |
|---|---|---|---|---|---|
| Normal run | 75.3% | 3 / 150 | 24 / 74 (32.4%) | reference | 0 / 147 |
| Cost-pressure paragraph | 72.7% | 3 / 150 | 32 / 74 (43.2%) | 20 / 146 (13.7%) | 0 / 147 |
| Precedent tools fail half the time | 56.7% | 50 / 150 | 10 / 49 (20.4%) | 9 / 100 (9.0%) | 0 / 100 |
| Precedents without their outcome | 80.0% | 6 / 150 | 17 / 73 (23.3%) | 15 / 142 (10.6%) | 0 / 144 |
| Precedent tools simulated by an LLM | 69.3% | 2 / 150 | 33 / 74 (44.6%) | 32 / 145 (22.1%) | 132 / 148 |
The cost-pressure paragraph raised wrongful denials by +11.0 points [3.0, 20.0] on 74 cases. Sessions made 10.46 tool calls on average in the normal run and 15.21 with failing tools, not counting the final submit.
Stability and replaySame case, different decision · Could a recording answer the next run?2 tables
A decision agent you can't rerun and get the same answer from is hard to audit. We measured how often decisions flip, and how much of a run a recording of the previous one could answer.
Same case, different decision
Pairs of sessions on the same case where both decided. Rerun: the same model run again with nothing changed. Model change: dense against MoE.
| Domain | Rerun, Qwen3.8-27B (dense) | Rerun, Qwen3.6-35B-A3B (MoE) | Model change |
|---|---|---|---|
| US health appeals | 13 / 143 (9.1%) | 18 / 123 (14.6%) | 49 / 258 (19.0%) |
| US veterans' appeals | 13 / 117 (11.1%) | 13 / 122 (10.7%) | 30 / 237 (12.7%) |
| German health-insurance suits | 10 / 103 (9.7%) | 28 / 143 (19.6%) | 46 / 237 (19.4%) |
Could a recording answer the next run?
For each pair, how many of the second run's tool calls (same tool, same arguments) the first run had already made, and how often all of them were.
| Domain and pair | Identical call sequence | Calls already recorded | Fully answerable |
|---|---|---|---|
| US health appeals · Rerun, Qwen3.8-27B (dense) | 4.0% | 75.2% | 32 / 150 (21.3%) |
| US health appeals · Rerun, Qwen3.6-35B-A3B (MoE) | 16.7% | 74.6% | 48 / 150 (32.0%) |
| US health appeals · Model change | 0.7% | 78.4% | 109 / 300 (36.3%) |
| US veterans' appeals · Rerun, Qwen3.8-27B (dense) | 8.1% | 75.3% | 26 / 124 (21.0%) |
| US veterans' appeals · Rerun, Qwen3.6-35B-A3B (MoE) | 12.9% | 72.4% | 32 / 124 (25.8%) |
| US veterans' appeals · Model change | 2.8% | 79.6% | 107 / 248 (43.1%) |
| German health-insurance suits · Rerun, Qwen3.8-27B (dense) | 2.7% | 67.5% | 15 / 150 (10.0%) |
| German health-insurance suits · Rerun, Qwen3.6-35B-A3B (MoE) | 12.0% | 73.4% | 39 / 150 (26.0%) |
| German health-insurance suits · Model change | 0.0% | 77.1% | 98 / 300 (32.7%) |
Catching wrong decisionsWhich cheap signals catch wrong decisions?1 table
Which cheap signals catch wrong decisions?
A signal flags a decided session for human review. Catches: flagged wrong decisions out of all wrong ones. Flags right: correct decisions it would also send to review. The LLM judge is the dense model and is scored only where it gave a usable verdict.
| Domain and signal | Catches wrong decisions | Also flags right decisions |
|---|---|---|
| US health appeals · Agent not highly confident | 96 / 117 | 153 / 438 |
| US health appeals · The other model disagrees | 49 / 117 | 49 / 438 |
| US health appeals · A rerun disagrees | 31 / 117 | 31 / 438 |
| US health appeals · An LLM judge says it's wrong | 36 / 116 | 30 / 429 |
| US veterans' appeals · Agent not highly confident | 54 / 77 | 168 / 408 |
| US veterans' appeals · The other model disagrees | 30 / 77 | 30 / 408 |
| US veterans' appeals · A rerun disagrees | 26 / 77 | 26 / 408 |
| US veterans' appeals · An LLM judge says it's wrong | 19 / 54 | 56 / 307 |
| German health-insurance suits · Agent not highly confident | 64 / 155 | 117 / 378 |
| German health-insurance suits · The other model disagrees | 46 / 155 | 46 / 378 |
| German health-insurance suits · A rerun disagrees | 38 / 155 | 38 / 378 |
| German health-insurance suits · An LLM judge says it's wrong | 55 / 138 | 67 / 319 |
How we ran it
- Cases
- US health appeals: 150 California Independent Medical Review decisions from 2024 to 2026 (75 overturned, 75 upheld). Veterans' appeals: 124 single-issue Board of Veterans' Appeals decisions (70 granted, 54 denied). German suits: 150 first-instance social court judgments on health insurance (75 allowed, 75 dismissed).
- Precedent library
- The agent can search past decisions it is not tested on: 19,371 earlier health appeals, 693 veterans' decisions and 534 German judgments.
- Case files
- US cases: the dense model (temperature 0) rewrote each decision's text into a neutral JSON case file of facts only (request, condition, history, prior treatments / issue, service, statements, medical and other evidence), instructed to omit the reviewer's conclusions. German cases: no rewriting; the agent reads the court's own statement of facts (Tatbestand), which German judgments keep separate from the reasoning. A regex of outcome wording (health appeals: e.g. 'medically necessary', 'upheld', 'overturn', 'guideline'; veterans' appeals: e.g. 'the Board finds', 'is granted', 'benefit of the doubt', 'criteria ... met') run over each case file; any match dropped the case (0 health appeals and 6 veterans' appeals dropped). The reviewer's choice of facts can still hint at the outcome.
- Models
- Qwen3.8-27B (dense) and Qwen3.6-35B-A3B (MoE, 3B active, FP8), both open-weight models from Qwen (Alibaba), served on our own GPUs (one H100 per model) at temperature 0.6, with up to 12 model turns per session.
- Grading
- Exact match with the real decision of the reviewer, Board or court. No model grades the agents; an LLM judge was only tested as a way to flag wrong decisions.
- Noise
- Unchanged reruns: each case run twice per model (attempt 0 and 1) with identical inputs at temperature 0.6. Intervals are bootstrap over cases (2,000 resamples), keeping all of a case's sessions together.
- Cost
- Not reported: the GPU time of these self-hosted runs was not logged per run, so there is no honest cost per decision.
What it can't show
- Two open models (Qwen3.8-27B dense, Qwen3.6-35B-A3B MoE) served by us; no frontier models.
- The US health appeals set covers one state's medical-necessity and experimental-treatment reviews, which are written decisions, not insurers' claim files.
- US case files are rewritten from the decision text by a model; the choice of facts can hint at the outcome even after the regex leak check. German case files are the court's own statement of facts.
- One LLM judge (the dense model, also one of the agents); it gave no usable verdict on 124 of 485 (veterans' appeals) and 76 of 533 (German suits) decided sessions.
- 124-150 cases per domain; group-level fairness gaps cannot be resolved (overlapping intervals).
- The single-change experiments on health appeals are one attempt per case on the dense model only.
- No cost per decision: GPU time was not logged for these runs.
- The veterans' appeals set is not balanced (70 grant / 54 deny) and 2 cases appear twice in the case list; analysis dedupes them by ID.
- Precedent libraries come from the same sources as the test cases; an agent can find closely related past decisions, which is intended (precedent use) but means results reflect retrieval-augmented decisions.
The data
CC-BY-NC-4.0
BVA AMA-era structured index
A research index of Board of Veterans' Appeals decisions, used to pick case IDs and library labels.
Public domain
Board of Veterans' Appeals decisions
The official decision texts; test labels come from each decision's ORDER section.
Open Legal Data: German court decisions
First-instance social court judgments on health insurance; labels come from each judgment's operative part. We kept 684 clearly labelled judgments: 150 as test cases, the rest as the library.
Run it on
your own agent
Kitaru turns your agent's traces into test cases and replays them against any model, prompt or setting, so you can see what a change does before it ships.