Apache-2.0
Snorkel AI: Multi Turn Insurance Underwriting
Multi-turn conversations between an underwriter and an AI assistant about small-business applications. We kept 252 of the 380 cases (see Method).
Which model should an underwriting agent run on?
12 models run the same insurance underwriting agent on 252 public cases. We measured what each correct decision costs, and how much of the gap between models is the prompt.
- Status
- Complete
- Run
- October 21, 2026 to October 22, 2026
- Models
- 12 (7 open-weight)
- Cases
- 252 (168 held out)
- Data
- Snorkel AI, Apache-2.0
- Spend
- $259.15
The result
Cost per correct decision, models tied at the top · lower is better - Claude Haiku 5.5
- $0.0038
- GLM 5.3 Flash
- $0.0092
- GPT-6.1 Sol
- $0.0178
- DeepSeek V4.1 Flash
- $0.0181
- Claude Sonnet 5.5
- $0.0273
- DeepSeek V4 Pro
- $0.0327
- Claude Opus 5.5
- $0.0512
- GLM 5.3
- $0.0737
- 01
Under the first prompt, accuracy ran from 59.5% to 97.6% and cost per correct decision varied 43× from cheapest to most expensive. Most of that gap was prompt fit, not underwriting knowledge.
- 02
One rewritten prompt, shared unchanged, lifted 10 of the 11 other models. 8 of 12 then scored between 97.6% and 99.4%, within the 2.4-point run-to-run noise of each other.
- 03
With accuracy tied, the choice is cost: Claude Haiku 5.5 got 166 / 168 right at $0.0038 per correct decision, against Claude Opus 5.5's 164 / 168 at $0.0512.
Results
Read the shared-prompt table first: it compares models under the best prompt we found, which is the fair comparison. The first-prompt table shows how different the picture looks before that.
One shared prompt, held-out cases
Sonnet 5.5's rewritten prompt, unchanged, on every model. 168 held-out cases the rewrite never saw. Highlighted rows are within the 2.4-point run-to-run noise of the top score.
| Model | Weights | Correct | Accuracy | 95% CI | Cost per correct | vs. first prompt (points) |
|---|---|---|---|---|---|---|
| GLM 5.3 Flash † | open | 167 / 168 | 99.4% | [96.7, 99.9] | $0.0092 | +7.7 [3.6, 12.5] |
| Claude Sonnet 5.5 | closed | 167 / 168 | 99.4% | [96.7, 99.9] | $0.0273 | +10.1 [5.4, 14.9] |
| Claude Haiku 5.5 | closed | 166 / 168 | 98.8% | [95.8, 99.7] | $0.0038 | +13.1 [8.3, 18.5] |
| DeepSeek V4 Pro † | open | 166 / 168 | 98.8% | [95.8, 99.7] | $0.0327 | +1.8 [−0.6, 4.8] |
| GPT-6.1 Sol | closed | 165 / 168 | 98.2% | [94.9, 99.4] | $0.0178 | +16.1 [10.1, 22.0] |
| DeepSeek V4.1 Flash † | open | 165 / 168 | 98.2% | [94.9, 99.4] | $0.0181 | +6.5 [2.4, 11.3] |
| GLM 5.3 † | open | 165 / 168 | 98.2% | [94.9, 99.4] | $0.0737 | +3.6 [0.6, 6.5] |
| Claude Opus 5.5 | closed | 164 / 168 | 97.6% | [94.0, 99.1] | $0.0512 | +0.6 [−3.0, 4.2] |
| MiniMax-M3 † | open | 162 / 168 | 96.4% | [92.4, 98.4] | $0.0182 | +11.9 [6.5, 17.3] |
| GPT-6 Luna | closed | 152 / 168 | 90.5% | [85.1, 94.0] | $0.0019 | +6.5 [0.6, 12.5] |
| gpt-oss-120b † | open | 146 / 168 | 86.9% | [81.0, 91.2] | $0.0115 | +31.0 [22.0, 39.9] |
| Qwen3.8 27B † | open | 139 / 168 | 82.7% | [76.3, 87.7] | $0.0533 | −0.6 [−7.7, 6.5] |
† Open-weight model on a commercial inference provider; price from list prices on 2026-10-19, unverified. Closed models use the vendor's list price on the same date. Cost per correct decision counts only the agent's model calls, not the simulated underwriter or the judges.
What else we found
- 04
The prompt beat every model swap: Sonnet 5.5 went from 89.3% to 99.4% on its own rewritten prompt, while running it twice with nothing changed moved it 2.4 points.
- 05
A cheap step-level screen ruled out clear failures but ranked the cheapest model that held quality 11th of 11. Full runs decided.
Full results
Every table behind the findings, grouped by question. Each opens in place.
More resultsFirst prompt, all cases · Does tuning each model's own prompt help? · Reasoning effort3 tables
First prompt, all cases
One system prompt written once, the same for every model, default settings, all 252 cases. The leaderboard most teams would stop at.
| Model | Weights | Correct | Accuracy | 95% CI | Cost per correct | Model calls per case |
|---|---|---|---|---|---|---|
| DeepSeek V4 Pro † | open | 246 / 252 | 97.6% | [94.9, 98.9] | $0.0355 | 9.12 |
| Claude Opus 5.5 | closed | 245 / 252 | 97.2% | [94.4, 98.7] | $0.0526 | 6.53 |
| GLM 5.3 † | open | 239 / 252 | 94.8% | [91.4, 97.0] | $0.0825 | 10.67 |
| GLM 5.3 Flash † | open | 233 / 252 | 92.5% | [88.5, 95.1] | $0.0097 | 10.98 |
| DeepSeek V4.1 Flash † | open | 230 / 252 | 91.3% | [87.1, 94.2] | $0.0210 | 9.18 |
| Claude Sonnet 5.5 | closed | 224 / 252 | 88.9% | [84.4, 92.2] | $0.0406 | 8.34 |
| Claude Haiku 5.5 | closed | 220 / 252 | 87.3% | [82.6, 90.9] | $0.0050 | 8.85 |
| GPT-6 Luna | closed | 219 / 252 | 86.9% | [82.2, 90.5] | $0.0019 | 7.87 |
| MiniMax-M3 † | open | 218 / 252 | 86.5% | [81.7, 90.2] | $0.0175 | 9.95 |
| Qwen3.8 27B † | open | 212 / 252 | 84.1% | [79.1, 88.1] | $0.0371 | 10.39 |
| GPT-6.1 Sol | closed | 211 / 252 | 83.7% | [78.7, 87.8] | $0.0201 | 7.09 |
| gpt-oss-120b † | open | 150 / 252 | 59.5% | [53.4, 65.4] | $0.0099 | 8.56 |
† Open-weight model on a commercial inference provider; price from list prices on 2026-10-19, unverified. Closed models use the vendor's list price on the same date. Cost per correct decision counts only the agent's model calls, not the simulated underwriter or the judges.
Does tuning each model's own prompt help?
Held-out accuracy for the models whose own rewritten prompt improved their learning score.
| Model | First prompt | Own rewritten prompt | Shared prompt |
|---|---|---|---|
| Claude Sonnet 5.5 | 89.3% | 99.4% | 99.4% |
| DeepSeek V4 Pro † | 97.0% | 98.2% | 98.8% |
| Claude Opus 5.5 | 97.0% | 98.2% | 97.6% |
| GPT-6.1 Sol | 82.1% | 97.6% | 98.2% |
| Claude Haiku 5.5 | 85.7% | 97.6% | 98.8% |
| GLM 5.3 Flash † | 91.7% | 97.6% | 99.4% |
| gpt-oss-120b † | 56.0% | 76.2% | 86.9% |
Reasoning effort
First prompt, held-out cases. In brackets: the change in points against the same model at its default effort.
| Model | Default | Low | Medium | High |
|---|---|---|---|---|
| Claude Opus 5.5 | 97.0% | 93.5% (−3.6) | not run | 97.6% (+0.6) |
| GPT-6.1 Sol | 82.1% | 85.0% (+3.0) | not run | 91.7% (+9.5) |
| GPT-6 Luna | 83.9% | not run | not run | 84.9% (+1.2) |
| Claude Haiku 5.5 | 85.7% | not run | not run | 84.5% (−1.2) |
| Claude Sonnet 5.5 | 89.3% | 72.6% (−16.7) | 81.5% (−7.7) | not run |
By question typeFirst prompt, all cases · Shared prompt, held-out cases2 tables
The agent answers six kinds of underwriting question. Under the first prompt, policy limits and product recommendations separated the field; the shared prompt closed most of those gaps.
First prompt, all cases
Correct answers per question type, all 252 cases.
| Model | Appetite Check | Product Recommendations | Small business eligibility | Business Classification | Policy Limits | Deductibles |
|---|---|---|---|---|---|---|
| DeepSeek V4 Pro † | 85 / 85 | 68 / 72 | 47 / 48 | 17 / 17 | 16 / 17 | 13 / 13 |
| Claude Opus 5.5 | 85 / 85 | 69 / 72 | 47 / 48 | 17 / 17 | 15 / 17 | 12 / 13 |
| GLM 5.3 † | 85 / 85 | 68 / 72 | 45 / 48 | 17 / 17 | 11 / 17 | 13 / 13 |
| GLM 5.3 Flash † | 82 / 85 | 65 / 72 | 46 / 48 | 17 / 17 | 11 / 17 | 12 / 13 |
| DeepSeek V4.1 Flash † | 83 / 85 | 69 / 72 | 42 / 48 | 17 / 17 | 9 / 17 | 10 / 13 |
| Claude Sonnet 5.5 | 81 / 85 | 56 / 72 | 44 / 48 | 14 / 17 | 16 / 17 | 13 / 13 |
| Claude Haiku 5.5 | 76 / 85 | 68 / 72 | 39 / 48 | 15 / 17 | 13 / 17 | 9 / 13 |
| GPT-6 Luna | 79 / 85 | 59 / 72 | 44 / 48 | 12 / 17 | 12 / 17 | 13 / 13 |
| MiniMax-M3 † | 79 / 85 | 54 / 72 | 45 / 48 | 17 / 17 | 11 / 17 | 12 / 13 |
| Qwen3.8 27B † | 78 / 85 | 48 / 72 | 42 / 48 | 17 / 17 | 14 / 17 | 13 / 13 |
| GPT-6.1 Sol | 60 / 85 | 65 / 72 | 44 / 48 | 14 / 17 | 15 / 17 | 13 / 13 |
| gpt-oss-120b † | 73 / 85 | 22 / 72 | 23 / 48 | 14 / 17 | 6 / 17 | 12 / 13 |
Darker cells missed more cases of that type.
Shared prompt, held-out cases
Correct answers per question type, 168 held-out cases.
| Model | Appetite Check | Product Recommendations | Small business eligibility | Business Classification | Policy Limits | Deductibles |
|---|---|---|---|---|---|---|
| GLM 5.3 Flash † | 57 / 57 | 48 / 48 | 32 / 32 | 11 / 11 | 10 / 11 | 9 / 9 |
| Claude Sonnet 5.5 | 57 / 57 | 47 / 48 | 32 / 32 | 11 / 11 | 11 / 11 | 9 / 9 |
| Claude Haiku 5.5 | 57 / 57 | 47 / 48 | 31 / 32 | 11 / 11 | 11 / 11 | 9 / 9 |
| DeepSeek V4 Pro † | 57 / 57 | 47 / 48 | 32 / 32 | 11 / 11 | 10 / 11 | 9 / 9 |
| GPT-6.1 Sol | 56 / 57 | 46 / 48 | 32 / 32 | 11 / 11 | 11 / 11 | 9 / 9 |
| DeepSeek V4.1 Flash † | 56 / 57 | 47 / 48 | 32 / 32 | 11 / 11 | 10 / 11 | 9 / 9 |
| GLM 5.3 † | 57 / 57 | 47 / 48 | 31 / 32 | 11 / 11 | 10 / 11 | 9 / 9 |
| Claude Opus 5.5 | 56 / 57 | 45 / 48 | 32 / 32 | 11 / 11 | 11 / 11 | 9 / 9 |
| MiniMax-M3 † | 57 / 57 | 42 / 48 | 32 / 32 | 11 / 11 | 11 / 11 | 9 / 9 |
| GPT-6 Luna | 56 / 57 | 33 / 48 | 32 / 32 | 11 / 11 | 11 / 11 | 9 / 9 |
| gpt-oss-120b † | 47 / 57 | 44 / 48 | 28 / 32 | 10 / 11 | 8 / 11 | 9 / 9 |
| Qwen3.8 27B † | 48 / 57 | 34 / 48 | 31 / 32 | 10 / 11 | 8 / 11 | 8 / 9 |
Darker cells missed more cases of that type.
FailuresHow the wrong answers went wrong1 table
Some models failed on the conversation, not the underwriting: they put their final answer on a question, sent tool calls that couldn't be parsed, or ran out of steps.
How the wrong answers went wrong
First prompt, all 252 cases. Unreadable tool calls overlap the other columns: the agent got an error and could carry on.
| Model | Ended on a question | Empty final answer | Wrong decision | Ran out of steps | Ran out of turns | Stopped replying | Unreadable tool call |
|---|---|---|---|---|---|---|---|
| DeepSeek V4 Pro † | 0 | 0 | 6 | 0 | 0 | 0 | 0 |
| Claude Opus 5.5 | 1 | 0 | 6 | 0 | 0 | 0 | 0 |
| GLM 5.3 † | 1 | 0 | 12 | 0 | 0 | 0 | 0 |
| GLM 5.3 Flash † | 2 | 0 | 16 | 1 | 0 | 0 | 0 |
| DeepSeek V4.1 Flash † | 1 | 0 | 21 | 0 | 0 | 0 | 0 |
| Claude Sonnet 5.5 | 5 | 0 | 23 | 0 | 0 | 0 | 0 |
| Claude Haiku 5.5 | 2 | 0 | 25 | 0 | 5 | 0 | 0 |
| GPT-6 Luna | 1 | 0 | 32 | 0 | 0 | 0 | 0 |
| MiniMax-M3 † | 1 | 0 | 32 | 0 | 1 | 0 | 0 |
| Qwen3.8 27B † | 0 | 0 | 23 | 17 | 0 | 0 | 0 |
| GPT-6.1 Sol | 15 | 5 | 20 | 0 | 0 | 1 | 0 |
| gpt-oss-120b † | 17 | 0 | 83 | 0 | 2 | 0 | 22 |
Testing a swap with KitaruStep-level screen vs. full runs · Full replays in Kitaru2 tables
Most teams don't run a benchmark; they ask whether one swap is safe. We tested that path from the Sonnet 5.5 agent with the first prompt: a cheap step-level screen first, then full replays.
Step-level screen vs. full runs
250 recorded Sonnet 5.5 model calls resent to each candidate. "Same tool" is how often a candidate's next step picks Sonnet's tool; Sonnet matches its own recording 58.3% of the time. The full-run column is each model's first-prompt accuracy on all 252 cases, paired against Sonnet.
| Candidate | Same tool as Sonnet | Full run vs. Sonnet (points, 95% CI) |
|---|---|---|
| Claude Haiku 5.5 | 29.8% | −1.6 [−7.1, 4.0] |
| DeepSeek V4 Pro † | 25.6% | +8.7 [4.4, 13.1] |
| GLM 5.3 † | 25.1% | +6.0 [1.2, 10.7] |
| Claude Opus 5.5 | 24.4% | +8.3 [4.0, 12.7] |
| GLM 5.3 Flash † | 22.6% | +3.6 [−1.6, 8.7] |
| DeepSeek V4.1 Flash † | 20.8% | +2.4 [−2.8, 7.9] |
| GPT-6.1 Sol | 20.4% | −5.2 [−11.5, 1.2] |
| Qwen3.8 27B † | 16.7% | −4.8 [−11.1, 1.6] |
| MiniMax-M3 † | 16.7% | −2.4 [−7.9, 3.2] |
| gpt-oss-120b † | 11.9% | −29.4 [−36.1, −23.0] |
| GPT-6 Luna | 8.9% | −2.0 [−7.5, 3.6] |
Highlighted: GPT-6 Luna, the cheapest model that held quality in full runs (full-run accuracy at most 5 points below Sonnet 5.5). The screen ranked it 11th of 11.
Full replays in Kitaru
48 cases recorded with Sonnet 5.5 and the first prompt, replayed end to end with each candidate swapped in, paired on the same cases.
| Swap Sonnet 5.5 for | Correct | Sonnet, same cases | Difference (points, 95% CI) |
|---|---|---|---|
| DeepSeek V4 Pro † | 47 / 48 | 43 / 48 | +8.3 [2.1, 16.7] |
| GLM 5.3 † | 45 / 48 | 43 / 48 | +4.2 [−6.2, 14.6] |
| GPT-6 Luna | 41 / 48 | 43 / 48 | −4.2 [−16.7, 8.3] |
| Claude Haiku 5.5 | 39 / 47 | 42 / 47 | −6.4 [−19.1, 6.4] |
Claude Haiku 5.5 has n = 47: one baseline case did not record cleanly. Spend for these replays: $9.11.
How we ran it
- Dataset
- Snorkel AI, Multi-Turn Insurance Underwriting and Multi-Turn Insurance Underwriting Code-Gen (Apache-2.0), plus US Census Bureau 2022 NAICS codes and concordances (public domain) and SBA size standards as recovered from the published traces. The insurer's environment (database, appetite guide, guidelines) is not published; we rebuilt it from the traces. A rule-based oracle re-derives each reference answer from the rebuilt environment; the 128 cases where it disagrees with the published reference are dropped, so every scored case is answerable from data the agent can see. 252 of 380 cases kept.
- Split
- 84 learning cases and 168 held-out cases, stratified by question type. Prompt tuning only saw the learning cases; every tuned, shared-prompt, effort and repeat result is on the held-out cases.
- Agent
- Six tools over the insurer's rebuilt database and guidelines: list_tables, get_table_descriptions, get_table_schema, get_table_data_dictionary, read_query (read-only SQL over the rebuilt insurer database), get_underwriting_guidelines. 30 agent model calls and 12 underwriter turns per case; the case ends at a message containing 'FINAL ANSWER:'; an empty agent turn gets 'Please continue.' at most twice; tool results capped at 200 rows.
- Underwriter
- Claude Haiku 5.5 plays the underwriter: it holds the company's application details from the dataset, opens with the case's request, and answers the agent's questions from that data only. It does not know the answer. Its cost is reported apart from the agent's.
- Grading
- Two judges (GPT-6 Luna and Claude Haiku 5.5) compare each final answer with the reference. If the two judges disagree (or one fails to return a verdict), GPT-6.1 Sol decides. Each judge agrees with Snorkel's human labels 87.5% to 90.0% of the time (n = 160, 150, 157). Re-grading the three judges' own answers with other judges moved them +1.6, −0.8, −0.8 points.
- Noise
- Sonnet 5.5 run a second time on the 168 held-out cases with the first prompt and the same settings, paired against its first run. It moved 2.4 points [−3.6, 8.3]. Paired bootstrap over cases (resample case-level differences with replacement), percentile interval; Wilson score intervals for single accuracies. 4000 resamples. A difference is called a tie when its 95% interval includes 0. In prose we also treat differences under 3 points as noise, given the 2.4-point repeat difference.
- Prompt rewriting
- Claude Opus 5.5 read up to 12 failed learning cases (conversation tail, final answer, reference, grader reason) and rewrote the whole system prompt with general instructions. A rewrite was kept only if it scored higher on the 84 learning cases; at most 2 rounds. A rewrite was rejected and retried if it contained any number of 3+ digits not in the original prompt, or any company name from the learning cases. The held-out cases were never shown to the optimizer. What changed: Conversation discipline, not underwriting knowledge: ask which line of business the application is for before recommending others; classify the business from the database when no code is given and treat it as settled; commit to one answer; never end the case on a question; answer only what was asked.
- Memorization
- 24 cases whose answers depend on the insurer's synthetic appetite table, asked of every model with no tools and no data (company profile and question only). Product recommendations: 0 of 72 correct across models. Appetite checks: 41 of 216, nearly all lodging and lessor businesses that the guidelines decline for fire risk, which a model can reason out. No sign of memorization.
- Cost
- Cost per correct decision = the agent's model-call cost over all cases, divided by correct answers, at list prices on October 19, 2026. Failed and timed-out cases stay in the count as wrong.
What it can't show
- One run per case per model and condition; differences under about 3 points are within run-to-run noise.
- The insurer's environment is reconstructed from published traces; 128 of 380 cases were dropped because the rebuilt environment did not reproduce their reference answer.
- The underwriter is simulated (Claude Haiku 5.5), not a person.
- Grading is by LLM judges validated against Snorkel's labels at 87.5-90.0% agreement per judge (n=150-160); the combined panel was not validated as a unit.
- Three judges (GPT-6 Luna, Claude Haiku 5.5, GPT-6.1 Sol) are also candidates; re-grading by other judges moved them by -0.8 to +1.6 points.
- The first-run system prompt was written once and not tuned for any model; first-run rankings largely reflect prompt fit.
- Prompt tuning: one optimizer (Opus 5.5), at most 2 rounds, 84 learning cases, one held-out run per tuned prompt. Five models were not tuned on their own (GPT-6 Luna: no rewrite improved its learning score; GLM 5.3, DeepSeek V4.1 Flash, MiniMax-M3, Qwen3.8 27B: only the shared prompt was run).
- The shared prompt was tuned on a Claude model and may favor Claude models.
- Effort settings were only tested on the five closed models.
- Prices are list prices on 2026-10-19; open-weight prices are from an unverified table, and open-weight models ran on one commercial inference provider (prices and behavior vary by provider).
- The failure categories 'final_answer_asks_for_information', 'empty_final_answer' and 'wrong_decision' are a text-pattern classification of wrong answers, not a judged label.
- The Kitaru full replay covers 48 cases; most of its intervals are wide.
- The step-level screen and full replay both use the first prompt; we did not repeat them under the rewritten prompt.
The data
Apache-2.0
Snorkel AI: Multi Turn Insurance Underwriting Code Gen
Multi-turn conversations between an underwriter and an AI assistant about small-business applications. We kept 252 of the 380 cases (see Method).
Run it on
your own agent
Kitaru turns your agent's traces into test cases and replays them against any model, prompt or setting, so you can see what a change does before it ships.