Benchmarks

The best model depends on your agent.

We test models, and the infrastructure that serves them, on real agent work by industry, starting with insurance, and count what each correct answer costs. Every experiment publishes its method, its noise and what it can't show.

8 benchmarks · last updated October 29, 2026

Featured industry

Insurance

Underwriting, claims and appeals agents on public insurance data: who keeps an agent improving, which model to run, what each correct decision costs, and how often a rerun changes the answer.

  • Improving a Claims Agent

    Who keeps a claims agent improving, batch after batch?

    Gain over the agent left alone, 800 appeals · higher is better
    ZenML AI engineer
    +4.0 points
    Codex
    +0.7 points

    Agent improvement · Model switch · Public data

  • Underwriting Agent

    Which model should an underwriting agent run on?

    Cost per correct decision, models tied at the top · lower is better
    Claude Haiku 5.5
    $0.0038
    GLM 5.3 Flash
    $0.0092
    GLM 5.3
    $0.0737

    Model choice · Prompting · Cost · Public data

  • Appeals Decisions

    Can an agent decide real appeals?

    Accuracy against real outcomes, Qwen3.8-27B (dense) · higher is better
    Health appeals
    76.0%
    Veterans' appeals
    80.2%
    German health suits
    61.0%

    Reliability · Open models · Public data

Customer service and telecom

Support agents that take actions for customers: whether a model swap is safe, and how to tell before it ships.

  • tau2-bench Telecom

    Which model holds up as a telecom support agent?

    Tasks solved out of 40 · higher is better
    GLM-5.3
    39 / 40
    GPT-5.2 (baseline)
    27 / 40
    GPT-5.4 mini
    19 / 40

    Model swap · Customer service · Kitaru replay · Public data

  • SwapBench

    Is it safe to swap the model behind your agent?

    Right verdicts out of 10 swaps · higher is better
    Next-step agreement
    3 right, 7 wrong
    Full replay, 60 scenarios
    4 right, 0 wrong
    Full replay, 260 scenarios
    5 right, 0 wrong

    Model swap · Customer service · Kitaru replay

AI infrastructure

Self-hosted agents on GPUs, for any industry: how hardware, load and the serving engine change what gets solved and what it costs.

  • ML Training Ops

    Can a self-hosted agent run ML training operations?

    Hard-set solve rate · higher is better
    H100
    34 / 48
    H200
    45 / 48

    Self-hosting · GPUs · Coding agents · Stopped

  • GPU & Serving Settings

    Does the GPU change what a self-hosted agent solves?

    Terminal-Bench solve rate by GPU · higher is better
    H100
    29.9%
    H200
    45.4%
    B200
    49.1%

    Self-hosting · GPUs · Cost · Coding agents

  • SGLang vs vLLM

    Does the serving engine change what one GPU serves?

    Requests completed on the same traffic, one H100 · higher is better
    Stock vLLM
    504
    Stock SGLang
    277
    SGLang, radix cache off
    435

    Serving · Self-hosting · GPUs

How we run these benchmarks

The whole agent, not the model
A model's score depends on the prompt, effort setting and tools around it. We compare models under the best prompt we can write, not the first one.
Cost per correct answer
Token price is not what you pay. A cheap model that takes 11 calls a case, or fails half the time, costs more per correct answer.
Full runs decide
Checking single steps is a cheap way to rule models out. The verdict comes from running whole tasks end to end.
Measured noise
We run the same agent twice to see how far a score moves by chance. Anything inside that gap is a tie, and we call it one.
Checked grading
Graders are deterministic wherever the task allows. LLM judges are checked against human labels, and prompt changes are scored on held-out cases they were never tuned on.
Caveats in the open
Each write-up says what the test can't show: one domain, synthetic users, single runs, list prices. Results that don't flatter anyone stay in.
Who runs these
The ZenML team runs these benchmarks. Some use Kitaru, our open-source tool for testing agent changes, to record and replay agent runs. Kitaru is usually the bench, not a contestant. One benchmark does compare our own AI engineer with Codex at improving a claims agent; its write-up says plainly that the setup favours our way of working, and it publishes the earlier run where neither improver pulled ahead. We publish the findings that cut against easy advice too, like a cheap closed model tying the best open ones for less per correct answer.

Run it on
your own agent

Kitaru turns your agent's traces into test cases and replays them against any model, prompt or setting, so you can see what a change does before it ships.

Start free