California DMHC Independent Medical Review determinations
California's open data on independent reviews of health-plan denials. 800 appeals from 2025 and 2026 as test cases, and earlier ones as the precedent library.
Who keeps a claims agent improving, batch after batch?
A claims agent decides 800 real California health appeals in eight batches, and switches to a cheaper open model halfway. After each batch, two improvers may change it: our AI engineer, working through Kitaru, and Codex, given the plain request a team would make, "improve my agent". A third copy is never touched.
- Status
- Complete
- Run
- October 28, 2026 to October 29, 2026
- Appeals
- 800 in 8 batches
- Agent model
- gpt-6.1-sol, then Qwen3.8-27B
- Improvers
- Both on gpt-6-luna
- Grading
- Real ruling, exact match
The result
Gain over the agent left alone, 800 appeals · higher is better - ZenML AI engineer
- +4.0 points
- Codex
- +0.7 points
Balanced accuracy, paired appeal by appeal. Codex's gain is within the noise.
- 01
Over all 800 appeals, the agent our AI engineer improved scored 79.1% balanced accuracy, against 75.7% for the one Codex improved and 75.1% for the one left alone. Compared appeal by appeal, it was +3.3 points ahead of Codex's [0.9, 5.8] and +4.0 ahead of the untouched agent [1.1, 6.9].
- 02
Most of the gap opened after the switch to the open model, which on about 1 appeal in 10 ran out of tokens or model requests before it decided. Our AI engineer counted how every recorded session ended, found those stops and raised the limits. From batch 6 it left 2 of 300 appeals undecided, against 15 for Codex and 38 for the untouched agent.
- 03
Codex looked for the cause in the prompt. In four rounds it tested a prompt change, saw it score lower and sensibly shipped nothing, but it never raised the limits. Its agent ended level with the one nobody touched (+0.7 points, within the noise).
Score on each new batch
Every round, each agent decided a fresh batch of 100 appeals. Then 20 of those rulings were revealed to the improvers, and each had 45 minutes and 3 reruns to propose one change. A change shipped only if it scored at least as well as the current version on 20 more appeals per batch that no improver ever saw.
ZenML AI engineer: batch 1 79.4%, batch 2 82.2%, batch 3 90.0%, batch 4 66.9%, batch 5 74.4%, batch 6 86.2%, batch 7 84.2%, batch 8 77.4%. Codex: batch 1 82.1%, batch 2 82.9%, batch 3 90.7%, batch 4 66.4%, batch 5 65.7%, batch 6 81.4%, batch 7 78.7%, batch 8 66.7%. Switch to Qwen3.8-27B from batch 4.
- AI engineer vs Codex+3.3 [0.9, 5.8]
- AI engineer vs left alone+4.0 [1.1, 6.9]
- Codex vs left alone+0.7 [−2.5, 4.0]
Balanced accuracy on each new batch
Each batch is 100 appeals none of the agents had seen, decided by the version each line had shipped so far. Batches 1 to 3 ran on gpt-6.1-sol, batches 4 to 8 on Qwen3.8-27B.
| Batch | Model | ZenML AI engineer | Codex | Left alone |
|---|---|---|---|---|
| 1 | gpt-6.1-sol | 79.4% | 82.1% | 78.7% |
| 2 | gpt-6.1-sol | 82.2% | 82.9% | 77.7% |
| 3 | gpt-6.1-sol | 90.0% | 90.7% | 86.7% |
| 4 | Qwen3.8-27B | 66.9% | 66.4% | 69.1% |
| 5 | Qwen3.8-27B | 74.4% | 65.7% | 68.1% |
| 6 | Qwen3.8-27B | 86.2% | 81.4% | 78.3% |
| 7 | Qwen3.8-27B | 84.2% | 78.7% | 76.4% |
| 8 | Qwen3.8-27B | 77.4% | 66.7% | 67.9% |
| Batches 1 to 3 | gpt-6.1-sol | 84.2% | 85.6% | 81.3% |
| Batches 4 to 8 | Qwen3.8-27B | 76.9% | 71.3% | 71.7% |
| All 800 appeals | 95% CI below | 79.1% (75.9% to 82.3%) | 75.7% (72.4% to 78.8%) | 75.1% (71.8% to 78.3%) |
Balanced accuracy is the average of accuracy on overturned and on upheld appeals; an appeal left undecided counts as wrong. Intervals are bootstrap over appeals.
What else we found
- 04
On the open model alone, batches 4 to 8, the gap over Codex was +5.6 points [2.2, 9.3]. On the frontier model, batches 1 to 3, Codex's agent was slightly ahead (85.6% against 84.2%).
- 05
Approval on appeals no improver ever saw turned down three changes that looked good on the improvers' own tests: Codex's round 2 change and two of our AI engineer's. Without that check, a change fitted to the few known answers ships.
- 06
The comparison is uneven on purpose. Codex got only the shared instructions, the way a team without this know-how would run it. Our AI engineer ran the way we run it: with Kitaru's recorded sessions, and an instruction to count how every session ended before choosing what to study. A first run without that instruction found no difference between the three agents; the rule is now built into the AI engineer.
- 07
Improving is cheap; checking is not. Both improvers together cost $1.26 over all eight rounds. Running the agents and checking each change cost about $75 in GPU time and API calls.
Full results
Every table behind the findings, grouped by question. Each opens in place.
What changed, and what it cost in mistakesWhat each improver changed · Which mistakes each line made2 tables
What each improver changed
One proposed change per round. Scores are the approval rule's: average balanced accuracy over 3 runs on every held-out appeal so far, candidate against current version.
| Round | ZenML AI engineer | Codex |
|---|---|---|
| 1 | The same idea for residential addiction treatment (ASAM levels 3.1 and 3.5). Shipped · 0.833 vs 0.798 | Sparse residential-treatment summaries count as missing, not as evidence against. Shipped · 0.881 vs 0.798 |
| 2 | Widen the residential rule to all residential appeals. Shipped · 0.917 vs 0.906 | Don't infer necessity from a numeric score or an incidental finding alone. Not shipped · 0.917 vs 0.933 |
| 3 | Treat omissions as unknown in every review type. Not shipped · 0.819 vs 0.893 | Tested widening its residential rule to other treatments. Tested, no change |
| 4 | Output limit 4,096 to 8,192 tokens; request limit 12 to 16. Shipped · 0.775 vs 0.726 | Tested a counterevidence rule. Tested, no change |
| 5 | Request limit 16 to 20. Shipped · 0.756 vs 0.745 | Tested two prompt clarifications. Tested, no change |
| 6 | A rule for proton-beam therapy. Shipped · 0.786 vs 0.781 | At most three precedent searches per appeal. Shipped · 0.748 vs 0.726 |
| 7 | Autism therapy (ABA): dose and setting need patient-specific support. Shipped · 0.813 vs 0.788 | Tested a rule on patient-specific criteria. Tested, no change |
| 8 | Named organs at risk count as evidence for proton beam. Not shipped · 0.772 vs 0.822 | Limit the missing-summary rule to residential appeals. Shipped · 0.797 vs 0.765 |
Every change was a prompt instruction or a setting; neither improver changed the agent's code or tools. Round 8's changes were never scored on a new batch.
Which mistakes each line made
All 800 appeals. A wrongful denial is a denial the agent upheld that the independent reviewer overturned.
| Line | Wrongful denials | Wrongful approvals | Undecided | Undecided, batches 6 to 8 | Accuracy |
|---|---|---|---|---|---|
| ZenML AI engineer | 75 | 56 | 15 | 2 / 300 | 81.8% |
| Codex | 52 | 70 | 29 | 15 / 300 | 81.1% |
| Left alone | 133 | 29 | 54 | 38 / 300 | 73.0% |
Both improved lines wrongly denied far fewer appeals than the untouched one and wrongly approved more. Balanced accuracy weighs the two mistakes equally.
The first runThe first run, before the fixes1 table
We ran the same design once before. It showed no difference, and exposed two problems: approval scored changes on the appeals the improver had studied, and our AI engineer read the step that said to count every session's ending, then skipped it. We fixed both and ran again.
The first run, before the fixes
The same appeals, batches and models. Approval used the revealed appeals the improver had studied, and our AI engineer had no instruction to count how every session ended. No line beat another by more than the noise.
| Batch | ZenML AI engineer | Codex | Left alone |
|---|---|---|---|
| 1 | 79.4% | 80.1% | 81.2% |
| 2 | 77.7% | 78.4% | 78.4% |
| 3 | 85.3% | 90.0% | 86.7% |
| 4 | 64.8% | 64.5% | 65.4% |
| 5 | 71.2% | 73.7% | 76.6% |
| 6 | 80.2% | 85.2% | 78.3% |
| 7 | 80.0% | 74.2% | 72.2% |
| 8 | 74.8% | 75.0% | 66.9% |
| All 800 appeals | 75.6% (72.2% to 78.9%) | 76.8% (73.4% to 80.2%) | 75.5% (72.2% to 78.6%) |
Paired over all 800: ZenML AI engineer minus Codex −1.2 points [−4.0, 1.3]; every interval includes zero.
How we ran it
- Cases
- California Independent Medical Review appeals from 2025 and 2026: a health plan denied a treatment, the patient appealed, and an independent reviewer upheld or overturned the denial. Of 3,131 usable appeals in filing order, each batch draws 100 at random from its own window of time.
- The agent
- It reads a facts-only case file, searches a library of 19,374 earlier decisions (2016 to 2023), and submits a decision. It is the same claims agent as our Appeals Decisions benchmark.
- Three lines
- One copy is left alone. One is improved by our AI engineer, which reads the agent's recorded sessions and reruns it only through Kitaru experiments. One is improved by Codex, which reads the same recordings as files and reruns with a plain tool. Both improvers run on gpt-6-luna, with the same revealed rulings, budget (45 minutes, 3 reruns, 1 change per round) and network.
- Approval
- A fixed rule decides what ships. The candidate and the current version each run 3 times on every held-out appeal so far (20 per batch, never shown to an improver), and the candidate ships if its average is at least as good.
- Model switch
- Batches 1 to 3 run the claims agent on gpt-6.1-sol. Before batch 4 every line switches to Qwen3.8-27B, a cheaper open model served on our own GPUs, to see whether the improvers notice what breaks.
- Grading
- Exact match with the independent reviewer's real ruling. No model grades the agent. The score is balanced accuracy, so an agent cannot win by approving everything.
- Noise
- Differences are paired appeal by appeal, with 95% bootstrap intervals over appeals (2,000 draws). On batch 1, where all three lines ran the identical starting agent, they scored 78.7%, 82.1% and 79.4%: one batch of 100 swings by about 3 points with nothing changed.
- Cost
- About $77 for the second run: $45.61 of GPU time for the open model, about $31 of OpenAI usage by the frameworks' own estimates, $1.26 of it for both improvers. The OpenAI figure leaves out our AI engineer's Kitaru experiments. 11h 9m from the first batch to the last audit.
What it can't show
- One state's medical-necessity and experimental-treatment reviews: written decisions, not insurers' claim files.
- Case files are written by a model from decision texts that include the reviewer's reasoning. A check drops files with outcome wording, but the choice of facts can still hint at the outcome.
- The comparison is uneven on purpose: Codex as a team without our know-how would run it, our AI engineer as the ZenML team runs it. It does not say how Codex would do with an expert's instructions.
- Each design ran once. The first run, before we fixed what it exposed, found no difference between the three lines; a third run could rank them differently again.
- The intervals cover which appeals were drawn, not how differently an improver might act on another night.
- Balanced accuracy weighs a wrongful denial and a wrongful approval equally. An insurer, or a patient refused care, would not.
The data
Run it on
your own agent
Kitaru turns your agent's traces into test cases and replays them against any model, prompt or setting, so you can see what a change does before it ships.