MIT
tau2-bench (Sierra Research)
The open harness the agents ran in, and the retail world (catalogue, users and orders) the retail scenarios were generated on. SwapBench's own scenario sets are not released yet.
Is it safe to swap the model behind your agent?
13 model swaps behind two customer-service agents, a bank and a store. We ran each swap on hidden scenarios to learn the truth, then asked which cheaper checks would have caught the regressions.
- Status
- Complete
- Run
- October 13 to 15, 2026
- Domains
- Bank and retail support
- Swaps
- 13 (10 with a clear answer)
- Scenarios
- 1,000 (240 hidden per domain)
- Agent tokens
- $412.92
The result
Right verdicts out of 10 swaps · higher is better - Next-step agreement
- 3 right, 7 wrong
- Full replay, 60 scenarios
- 4 right, 0 wrong
- Full replay, 260 scenarios
- 5 right, 0 wrong
- 01
Next-step agreement, the cheap check, called 5 of 8 regressed swaps safe and called both swaps that hold (2 of 2) unsafe. It got 3 of 10 verdicts right.
- 02
A full replay on 260 scenarios never called a regression safe or a good swap unsafe. It got 5 verdicts right and said "not enough evidence" on the other 5.
- 03
Imitating the incumbent is not the same as doing the job: on retail, self-hosted Qwen3.8-27B FP8 matched 440 of 475 of the incumbent's next tool calls, yet resolved 9.8 points fewer hidden conversations.
Which checks can tell
Each swap was first run for real on hidden scenarios to learn whether it regressed. Then each check gave its verdict without seeing that answer.
How often each check got the verdict right
13 candidate swaps; 10 have a clear answer on the hidden set (8 regressed, 2 hold) and 3 are too close to call, so they are not scored.
| Check | Right verdicts | Called a regression safe | Called a safe swap unsafe | Not enough evidence |
|---|---|---|---|---|
| Next-step agreement | 3 of 10 | 5 of 8 | 2 of 2 | 0 |
| Full replay, 60 scenarios | 4 of 10 | 0 of 8 | 0 of 2 | 6 |
| Full replay, 260 scenarios | 5 of 10 | 0 of 8 | 0 of 2 | 5 |
A full replay can say it has not seen enough evidence; next-step agreement always gives a verdict.
What else we found
- 04
A three-line policy clarification turned the regressed self-hosted Qwen3.8-27B (−5.6 points) into a bank agent that holds (−1.3 [−4.0, 1.5]), confirmed on 200 scenarios nobody had looked at (−0.5). Tool guardrails alone left it at −5.4.
- 05
The same weights behind vLLM's generic tool parser resolved 1 of 240 retail conversations, with no errors raised. How the endpoint is configured is a swap risk of its own.
- 06
Fine-tuning Qwen3.8-27B on its own successful bank conversations did not change success (−4.4 against −5.6 before); output tokens per conversation fell from 2,341 to 2,047.
Full results
Every table behind the findings, grouped by question. Each opens in place.
More which checks can tellEvery swap, every check1 table
Every swap, every check
Truth is the paired success difference against the incumbent on the 240 hidden scenarios, twice each, with a 5-point margin; a swap whose harmful actions rise also counts as regressed. Next-step agreement compares the candidate's next tool call with the incumbent's recorded one, against the incumbent's agreement with itself.
| Swap | Domain | Truth (points, 95% CI) | Next-step agreement | Next-step verdict | Replay, 60 | Replay, 260 |
|---|---|---|---|---|---|---|
| GPT-5 mini | Bank | Holds +0.2 [−2.5, 2.9] | 84.9% vs 90.0% | Unsafe ✗ false alarm | Safe ✓ | Not enough evidence |
| GPT-6 Luna | Bank | Regressed −2.5 [−5.4, 0.4] | 87.6% vs 90.0% | Safe ✗ false safe | Not enough evidence | Not enough evidence |
| GPT-5.4 nano | Bank | Regressed −37.5 [−42.9, −32.1] | 75.3% vs 90.0% | Unsafe ✓ | Unsafe ✓ | Unsafe ✓ |
| Qwen3.8-27B BF16 | Bank | Regressed −5.6 [−8.8, −2.5] | 85.7% vs 90.0% | Safe ✗ false safe | Not enough evidence | Unsafe ✓ |
| Qwen3.8-27B FP8 | Bank | Regressed −6.0 [−9.4, −2.9] | 88.4% vs 90.0% | Safe ✗ false safe | Not enough evidence | Unsafe ✓ |
| Qwen3.6-35B-A3B MoE | Bank | Unclear −4.2 [−7.9, −0.4] | 85.3% vs 90.0% | Safe | Not enough evidence | Not enough evidence |
| Qwen3.8-27B + LoRA RFT | Bank | Regressed −4.4 [−7.3, −1.7] | 84.9% vs 90.0% | Unsafe ✓ | Not enough evidence | Not enough evidence |
| Claude Haiku 5.5 (re-run against itself) | Retail | Unclear −2.1 [−5.4, 1.3] | same model | Safe | Not enough evidence | Safe |
| Qwen3.8-27B FP8 | Retail | Regressed −9.8 [−13.8, −6.0] | 92.6% vs 97.3% | Safe ✗ false safe | Not enough evidence | Unsafe ✓ |
| Qwen3.6-35B-A3B MoE | Retail | Holds −0.6 [−3.3, 2.1] | 91.8% vs 97.3% | Unsafe ✗ false alarm | Safe ✓ | Not enough evidence |
| Qwen3.8-27B, thinking off | Retail | Unclear −2.5 [−5.4, 0.2] | 87.8% vs 97.3% | Unsafe | Safe | Not enough evidence |
| Qwen3.8-27B with vLLM's generic hermes tool parser | Retail | Regressed −95.0 [−97.1, −92.7] | 0.0% vs 97.3% | Unsafe ✓ | Unsafe ✓ | Unsafe ✓ |
| Qwen3.8-27B BF16 | Retail | Regressed −8.1 [−12.1, −4.4] | 92.4% vs 97.3% | Safe ✗ false safe | Not enough evidence | Not enough evidence |
✓ matches the truth. ✗ false safe: a regression called safe. ✗ false alarm: a swap that holds called unsafe. Swaps with unclear truth are not marked.
ResultsBank: full conversations on the hidden set · Retail: full conversations on the hidden set2 tables
The ground truth: every candidate against the incumbent, GPT-5 for the bank and Claude Haiku 5.5 for retail.
Bank: full conversations on the hidden set
240 hidden scenarios, twice each, against stateful tools and a simulated customer. Graded by the tools' end state; GPT-5 is the incumbent.
| Model | Resolved | Success | 95% CI | vs. GPT-5 (points, 95% CI) | Harmful actions per conversation | Cost per resolved | Truth |
|---|---|---|---|---|---|---|---|
| GPT-5 | 452 / 480 | 94.2% | [91.7, 96.5] | incumbent | 0.031 | $0.0769 | |
| GPT-5 mini | 453 / 480 | 94.4% | [92.1, 96.3] | +0.2 [−2.5, 2.9] | 0.048 | $0.0138 | Holds |
| Qwen3.8-27B + guardrails + 3-line policy clarification | 446 / 480 | 92.9% | [90.2, 95.4] | −1.3 [−4.0, 1.5] | 0.046 | self-hosted | Holds |
| GPT-6 Luna | 440 / 480 | 91.7% | [89.0, 94.2] | −2.5 [−5.4, 0.4] | 0.073 | $0.0011 | Regressed |
| Qwen3.6-35B-A3B MoE | 433 / 481 | 90.0% | [86.7, 92.9] | −4.2 [−7.9, −0.4] | 0.060 | self-hosted | Unclear |
| Qwen3.8-27B + LoRA RFT | 431 / 480 | 89.8% | [86.7, 92.7] | −4.4 [−7.3, −1.7] | 0.092 | self-hosted | Regressed |
| Qwen3.8-27B + tool guardrails | 426 / 480 | 88.8% | [85.2, 91.9] | −5.4 [−8.8, −2.3] | 0.098 | self-hosted | Regressed |
| Qwen3.8-27B BF16 | 425 / 480 | 88.5% | [85.0, 91.9] | −5.6 [−8.8, −2.5] | 0.096 | self-hosted | Regressed |
| Qwen3.8-27B FP8 | 395 / 445 | 88.1% | [84.6, 91.5] | −6.0 [−9.4, −2.9] | 0.098 | self-hosted | Regressed |
| GPT-5.4 nano | 272 / 480 | 56.7% | [51.5, 61.9] | −37.5 [−42.9, −32.1] | 0.454 | $0.0080 | Regressed |
Self-hosted models ran on our own vLLM servers, so they have no token price. API costs use list prices pinned on the run dates and leave out the simulated customer.
Retail: full conversations on the hidden set
240 hidden scenarios, twice each, against stateful tools and a simulated customer. Graded by the tools' end state; Claude Haiku 5.5 is the incumbent.
| Model | Resolved | Success | 95% CI | vs. Claude Haiku 5.5 (points, 95% CI) | Harmful actions per conversation | Cost per resolved | Truth |
|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5 | 458 / 480 | 95.4% | [93.3, 97.3] | incumbent | 0.013 | $0.0188 | |
| Qwen3.6-35B-A3B MoE | 455 / 480 | 94.8% | [92.1, 97.1] | −0.6 [−3.3, 2.1] | 0.004 | self-hosted | Holds |
| Claude Haiku 5.5 (re-run against itself) | 224 / 240 | 93.3% | [90.0, 96.3] | −2.1 [−5.4, 1.3] | 0.021 | $0.0191 | Unclear |
| Qwen3.8-27B, thinking off | 446 / 480 | 92.9% | [90.4, 95.2] | −2.5 [−5.4, 0.2] | 0.010 | self-hosted | Unclear |
| Qwen3.8-27B BF16 | 419 / 480 | 87.3% | [83.8, 90.6] | −8.1 [−12.1, −4.4] | 0.027 | self-hosted | Regressed |
| Qwen3.8-27B FP8 | 411 / 480 | 85.6% | [81.9, 89.0] | −9.8 [−13.8, −6.0] | 0.019 | self-hosted | Regressed |
| Qwen3.8-27B with vLLM's generic hermes tool parser | 1 / 240 | 0.4% | [0.0, 1.3] | −95.0 [−97.1, −92.7] | 0.000 | self-hosted | Regressed |
Self-hosted models ran on our own vLLM servers, so they have no token price. API costs use list prices pinned on the run dates and leave out the simulated customer.
Fixing a regressed swapThe fix, confirmed on scenarios nobody looked at · Where bank conversations went wrong2 tables
Stock Qwen3.8-27B regressed as a bank agent, mostly by handing customers to a person when it didn't need to. Guardrails in the tools didn't help; three lines of policy did.
The fix, confirmed on scenarios nobody looked at
The policy clarification was found on the hidden set, so it was re-tested on 200 pool scenarios that were never inspected.
| Model | Resolved | Success | vs. GPT-5 (points, 95% CI) | Harmful actions |
|---|---|---|---|---|
| GPT-5 | 187 / 200 | 93.5% | incumbent | 7 |
| Qwen3.8-27B BF16 | 175 / 200 | 87.5% | −6.0 [−10.5, −1.5] | 20 |
| Qwen3.8-27B + guardrails + 3-line policy clarification | 372 / 400 | 93.0% | −0.5 [−4.3, 3.5] | 13 |
The fixed agent ran each pool scenario twice, so it has twice as many conversations.
Where bank conversations went wrong
Bank conversations GPT-5 resolved and the candidate did not, classified at the first decision that went wrong.
| First wrong decision | GPT-6 Luna | Qwen3.8-27B BF16 | Qwen3.8-27B FP8 |
|---|---|---|---|
| Needless handoff | 22 | 25 | 21 |
| Wrong refund arguments | 1 | 9 | 10 |
| Identity never verified | 5 | 4 | 4 |
| Wrong escalation arguments | 0 | 2 | 5 |
| Wrong dispute arguments | 1 | 1 | 2 |
| Action not in the correct resolution | 2 | 0 | 0 |
| Wrong card freeze arguments | 0 | 1 | 0 |
Most lost conversations were needless handoffs: the agent offered a transfer to a person, the customer accepted, and the right resolution had none.
How we ran it
- Scenarios
- Two generated domains with ground truth by construction: a fictional bank with 7 tools and a written policy, and a store built on tau2-bench's retail world (15 tools). Each domain has 60 visible, 200 pool and 240 hidden scenarios. The hidden sets are not released.
- Agents
- GPT-5 is the bank incumbent and Claude Haiku 5.5 the retail incumbent. Candidates include closed API models and self-hosted open models on vLLM, plus variants of one model: quantized, with thinking off, with guardrails, fine-tuned, and behind a different tool parser.
- Customer
- Claude Haiku 5.5 plays the customer from each scenario's instructions in every run. Money-moving tool calls are recorded, never executed.
- Grading
- No LLM judge. A conversation passes if the tools' final state equals the state after the correct calls. A harmful action is a successful write that matches no correct write.
- Truth
- A swap regressed if its paired success difference is 5 points or more below the incumbent, or harmful actions rise; it holds if the lower end of its 95% interval is above −5 points and harm doesn't rise; otherwise the truth is unclear and the swap is not scored.
- Checks
- Next-step agreement: the candidate sees each recorded step of the incumbent's visible conversations and picks its next action; safe if its agreement is within 5 points of the incumbent's agreement with itself. Full replay: whole conversations on 60 or 260 scenarios with the same 5-point gate, which can also answer "not enough evidence".
- Noise
- Paired cluster bootstrap over scenarios, 10,000 resamples. The retail incumbent re-run against itself moved −2.1 points [−5.4, 1.3] on 240 scenarios.
- Scenario review
- A separate Claude agent with no access to the generators or grader reviewed 50 hidden scenarios: 39 pass, 7 minor issues, 4 fail. All 4 failures are one retail kind (cancel an already-delivered order, 50 of 240 hidden retail scenarios), kept so published numbers stay comparable.
- Cost
- Agent token cost at list prices pinned on the run dates, recorded on every conversation: $412.92 in total. It leaves out the simulated customer and GPU time for the self-hosted models. Run dates: October 13, 2026 to October 15, 2026.
What it can't show
- Simulated customers only: Claude Haiku 5.5 plays every customer, and a different simulator can shift success rates and verdicts. Verdicts were never compared with real customers.
- Ground truth by construction assumes a faithful customer; script breaks by the simulator corrupt the truth itself.
- Scenario quality was checked by a Claude reviewer on 50 hidden scenarios, not by a human. One retail kind (cancel an already-delivered order, 50 of 240 hidden retail scenarios) penalizes helpful agents; it was kept in the released numbers.
- Scale: two domains, one incumbent per domain, 13 scored candidate swaps (10 with decidable truth: 8 regressed, 2 non-inferior). Error rates of the checks have wide uncertainty (e.g. 0 of 8 vs 5 of 8 false safes); the planned exact intervals and paired tests were not run.
- Truth is itself estimated from 240 scenarios x 2; three candidates have AMBIGUOUS truth and are not scored. Pooled over 500 scenarios, GPT-5 mini also raises harmful actions (+0.026 per conversation), so its NON-INFERIOR label holds for hidden-set success only.
- The replay checks and the truth use the same simulator, harness and grader (not independent); an independent-simulator truth was planned but not run.
- The v1 hidden sets were generated from seeds in the repository history, so they guard against agents peeking during a run, not against deliberate gaming.
- Costs: API costs use list prices pinned on 2026-10-13/15, not invoices; the simulated customer is excluded; self-hosted costs are GPU time at one concurrency level, without engineering or operations.
- Self-hosted FP8 and MoE fintech runs combine one full trial with a partial second trial cut short by a restart (445 and 481 conversations).
- Next-step agreement verdicts depend on a threshold; one 'correct' verdict (qwen27b-rft, 84.9% vs an 85.0% cut-off) rests on a 0.1-point margin.
- Prior work already establishes the mechanisms (compounding per-step errors, simulator effects, provider variance); the contribution is measuring swap checks against ground-truth outcomes.
The data
Run it on
your own agent
Kitaru turns your agent's traces into test cases and replays them against any model, prompt or setting, so you can see what a change does before it ships.