Apache-2.0
Terminal-Bench 2.0
Terminal tasks with deterministic tests, run through the Harbor framework. We used 36 of its 89 tasks (see Method).
Does the GPU change what a self-hosted agent solves?
One open model served with vLLM, four agent harnesses, 36 Terminal-Bench and 60 GAIA tasks. We changed the GPU, the number of agents sharing it and the serving settings, and counted solved tasks and GPU cost per solve.
- Status
- Complete
- Run
- October 9, 2026
- Models
- Qwen3.8-27B and Qwen3.6-35B-A3B, self-hosted
- Tasks
- 36 Terminal-Bench, 60 GAIA
- Trials
- 3,744
- GPUs
- H100, H200, B200
The result
Terminal-Bench solve rate by GPU · higher is better - H100
- 29.9%
- H200
- 45.4%
- B200
- 49.1%
mini-swe-agent with 32 agents on one GPU, same model and tasks.
- 01
The GPU changed what the agent solved. At 32 agents, a B200 instead of an H100 raised terminus-2 from 41.2% to 55.6% and mini-swe-agent from 29.9% to 49.1%: paired differences of +14.3 [7.4, 22.7] and +19.2 [8.6, 30.6] points.
- 02
Load cost solved tasks. On one H100, terminus-2 solved 52.8% with 8 agents sharing the GPU and 22.2% with 64; against 32 agents, going to 64 was −19.0 [−29.2, −9.3] points.
- 03
Most unsolved trials ran out of time, not ideas. At 32 agents on an H100, 92 of mini-swe-agent's 101 unsolved Terminal-Bench trials hit the time limit while still issuing new actions.
What each change did
The cleanest read: every change against the same baseline, one H100 shared by 32 agents, paired task by task.
Terminal-Bench
36 tasks, each change paired with the baseline on the same tasks.
| Change from one H100 with 32 agents | terminus-2 (points, 95% CI) | mini-swe-agent (points, 95% CI) |
|---|---|---|
| B200 instead of H100 | +14.3 [7.4, 22.7] | +19.2 [8.6, 30.6] |
| H200 instead of H100 | +3.2 [−4.6, 10.7] | +15.5 [5.8, 25.7] |
| 16 agents instead of 32 | +3.2 [−2.3, 9.3] | +13.7 [5.8, 22.2] |
| FP8 KV cache | +8.8 [0.9, 16.7] | +7.6 [0.7, 14.6] |
| FP8 weights | +0.5 [−6.9, 8.3] | +10.4 [2.8, 18.8] |
| MoE model (3B active) | +1.4 [−11.1, 14.8] | +19.2 [4.2, 35.4] |
| Speculative decoding (MTP) | +2.5 [−4.2, 9.9] | +6.3 [0.0, 13.2] |
| Prefix caching off | +1.9 [−6.5, 10.2] | +4.9 [−2.1, 12.5] |
| Thinking off | +4.2 [−5.1, 13.9] | +9.9 [0.5, 20.1] |
| H200 with speculative decoding | +17.1 [8.8, 25.9] | +20.1 [6.3, 35.4] |
| 2× time limit | +5.1 [−1.9, 13.4] | +9.0 [2.3, 16.4] |
| 4× time limit | +6.0 [−6.9, 19.0] | +9.0 [−2.1, 21.5] |
| 8 agents instead of 32 | +11.6 [3.7, 20.4] | +20.1 [8.3, 33.3] |
| 48 agents instead of 32 | −7.9 [−14.3, −1.4] | +4.9 [−1.4, 11.8] |
| 64 agents instead of 32 | −19.0 [−29.2, −9.3] | −3.5 [−12.5, 4.9] |
Differences in solve rate, paired by task. Highlighted rows changed both harnesses with intervals that exclude zero; grey values have intervals that include it. Many comparisons were made, so a few intervals excluding zero are expected by chance.
What else we found
- 04
The faster GPU cost more per hour but less per solved task: mini-swe-agent's GPU cost per solved task was $0.1088 on the B200 against $0.1425 on the H100 at 32 agents.
- 05
The effect is not specific to coding: on GAIA, an H200 instead of an H100 changed terminus-2 by +13.3 [3.3, 25.0] points and mini-swe-agent by +13.3 [0.0, 26.7].
- 06
Of four pre-registered claims, one held on fresh data: time-out share rises with median seconds per agent step. The three claims that a setting affects one harness more than another were not confirmed.
Full results
Every table behind the findings, grouped by question. Each opens in place.
More what each change didGAIA1 table
GAIA
60 tasks, one run per cell.
| Change from one H100 with 32 agents | terminus-2 (points, 95% CI) | mini-swe-agent (points, 95% CI) |
|---|---|---|
| H200 instead of H100 | +13.3 [3.3, 25.0] | +13.3 [0.0, 26.7] |
| 16 agents instead of 32 | +3.3 [−5.0, 13.3] | +8.3 [−3.3, 20.0] |
Differences in solve rate, paired by task. Highlighted rows changed both harnesses with intervals that exclude zero; grey values have intervals that include it. Many comparisons were made, so a few intervals excluding zero are expected by chance.
ResultsLoad: agents sharing one H100 · Hardware and serving settings at 32 agents · Four agent harnesses, same server · GAIA web research4 tables
Solve rates with task-level 95% intervals, and what the GPU time cost per solved task.
Load: agents sharing one H100
vLLM defaults on one H100, Terminal-Bench (36 tasks). Every trial counts, including ones that never started.
| Harness | Agents per GPU | Solved | Solve rate | 95% CI | GPU cost per solved task |
|---|---|---|---|---|---|
| terminus-2 | 8 | 19 / 36 | 52.8% | [36.1, 69.4] | $0.2086 |
| terminus-2 | 16 | 48 / 108 | 44.4% | [29.6, 59.3] | $0.1439 |
| terminus-2 | 32 | 89 / 216 | 41.2% | [27.8, 54.6] | $0.0952 |
| terminus-2 | 48 | 24 / 72 | 33.3% | [20.8, 47.2] | $0.0900 |
| terminus-2 | 64 | 16 / 72 | 22.2% | [11.1, 34.7] | $0.1358 |
| mini-swe-agent | 8 | 18 / 36 | 50.0% | [33.3, 66.7] | $0.2276 |
| mini-swe-agent | 16 | 47 / 108 | 43.5% | [28.7, 58.3] | $0.1495 |
| mini-swe-agent | 32 | 43 / 144 | 29.9% | [18.1, 43.1] | $0.1425 |
| mini-swe-agent | 48 | 25 / 72 | 34.7% | [20.8, 48.6] | $0.1009 |
| mini-swe-agent | 64 | 19 / 72 | 26.4% | [15.3, 38.9] | $0.1097 |
Hardware and serving settings at 32 agents
One change at a time from vLLM defaults on an H100, Terminal-Bench (36 tasks). Cells with more runs pool every run.
| Harness | Setting | GPU | Solved | Solve rate | 95% CI | GPU cost per solved task |
|---|---|---|---|---|---|---|
| terminus-2 | vLLM defaults | H100 | 89 / 216 | 41.2% | [27.8, 54.6] | $0.0952 |
| terminus-2 | H200 instead of H100 | H200 | 48 / 108 | 44.4% | [29.6, 59.3] | $0.0836 |
| terminus-2 | B200 instead of H100 | B200 | 60 / 108 | 55.6% | [41.7, 69.4] | $0.0970 |
| terminus-2 | H200 with speculative decoding | H200 | 21 / 36 | 58.3% | [41.7, 75.0] | $0.0626 |
| terminus-2 | Speculative decoding (MTP) | H100 | 63 / 144 | 43.8% | [31.9, 56.3] | $0.0859 |
| terminus-2 | FP8 KV cache | H100 | 36 / 72 | 50.0% | [36.1, 63.9] | $0.0842 |
| terminus-2 | FP8 weights | H100 | 30 / 72 | 41.7% | [27.8, 55.6] | $0.0993 |
| terminus-2 | MoE model (3B active) | H100 | 46 / 108 | 42.6% | [29.6, 56.5] | $0.0843 |
| terminus-2 | Prefix caching off | H100 | 31 / 72 | 43.1% | [29.2, 58.3] | $0.0952 |
| terminus-2 | Thinking off | H100 | 49 / 108 | 45.4% | [29.6, 61.1] | $0.0820 |
| terminus-2 | 2× time limit | H100 | 50 / 108 | 46.3% | [32.4, 60.2] | $0.1428 |
| terminus-2 | 4× time limit | H100 | 17 / 36 | 47.2% | [30.6, 63.9] | $0.2545 |
| mini-swe-agent | vLLM defaults | H100 | 43 / 144 | 29.9% | [18.1, 43.1] | $0.1425 |
| mini-swe-agent | H200 instead of H100 | H200 | 49 / 108 | 45.4% | [30.6, 60.2] | $0.0912 |
| mini-swe-agent | B200 instead of H100 | B200 | 53 / 108 | 49.1% | [34.3, 63.9] | $0.1088 |
| mini-swe-agent | H200 with speculative decoding | H200 | 18 / 36 | 50.0% | [33.3, 66.7] | $0.0908 |
| mini-swe-agent | Speculative decoding (MTP) | H100 | 52 / 144 | 36.1% | [23.6, 49.3] | $0.1203 |
| mini-swe-agent | FP8 KV cache | H100 | 27 / 72 | 37.5% | [22.2, 52.8] | $0.1106 |
| mini-swe-agent | FP8 weights | H100 | 29 / 72 | 40.3% | [26.4, 55.6] | $0.0987 |
| mini-swe-agent | MoE model (3B active) | H100 | 53 / 108 | 49.1% | [34.3, 63.9] | $0.0723 |
| mini-swe-agent | Prefix caching off | H100 | 25 / 72 | 34.7% | [20.8, 50.0] | $0.1268 |
| mini-swe-agent | Thinking off | H100 | 43 / 108 | 39.8% | [25.0, 53.7] | $0.0905 |
| mini-swe-agent | 2× time limit | H100 | 42 / 108 | 38.9% | [25.0, 53.7] | $0.1911 |
| mini-swe-agent | 4× time limit | H100 | 14 / 36 | 38.9% | [22.2, 55.6] | $0.3281 |
GPU cost per solved task = GPU run time while the agents ran (including tool waits and failed tasks, not server startup) × list price per GPU-hour, divided by solved tasks. List prices, October 2026: H100 $3.95, H200 $4.54, B200 $6.25 per hour. Sandbox cost is not included: we only have an estimate of it.
Four agent harnesses, same server
vLLM defaults on one H100, Terminal-Bench. opencode and pi ran one 36-task run per cell, too few for firm claims.
| Harness | Agents per GPU | Solved | Solve rate | 95% CI | GPU cost per solved task |
|---|---|---|---|---|---|
| terminus-2 | 16 | 48 / 108 | 44.4% | [29.6, 59.3] | $0.1439 |
| terminus-2 | 32 | 89 / 216 | 41.2% | [27.8, 54.6] | $0.0952 |
| mini-swe-agent | 16 | 47 / 108 | 43.5% | [28.7, 58.3] | $0.1495 |
| mini-swe-agent | 32 | 43 / 144 | 29.9% | [18.1, 43.1] | $0.1425 |
| opencode | 16 | 18 / 36 | 50.0% | [33.3, 66.7] | $0.1356 |
| opencode | 32 | 16 / 36 | 44.4% | [30.6, 61.1] | $0.0894 |
| pi | 16 | 15 / 36 | 41.7% | [25.0, 58.3] | $0.1599 |
| pi | 32 | 14 / 36 | 38.9% | [25.0, 55.6] | $0.1002 |
GAIA web research
60 GAIA tasks stratified by level, 10-minute limit, one run per cell.
| Harness | GPU | Agents per GPU | Solved | Solve rate | 95% CI | GPU cost per solved task |
|---|---|---|---|---|---|---|
| mini-swe-agent | H100 | 16 | 18 / 60 | 30.0% | [18.3, 41.7] | $0.1428 |
| mini-swe-agent | H100 | 32 | 13 / 60 | 21.7% | [11.7, 31.7] | $0.1126 |
| mini-swe-agent | H200 | 32 | 21 / 60 | 35.0% | [23.3, 46.7] | $0.0801 |
| terminus-2 | H100 | 16 | 14 / 60 | 23.3% | [13.3, 33.3] | $0.1701 |
| terminus-2 | H100 | 32 | 12 / 60 | 20.0% | [10.0, 30.0] | $0.1144 |
| terminus-2 | H200 | 32 | 20 / 60 | 33.3% | [21.7, 45.0] | $0.0788 |
FailuresWhy trials went unsolved as load rises1 table
Each unsolved trial was classified from its trajectory. The clock dominated: agents that were still making progress ran out of time, and more so as the GPU got busier.
Why trials went unsolved as load rises
vLLM defaults on one H100, Terminal-Bench. Unsolved trials by cause; most hit the time limit while the agent was still issuing new actions.
| Harness | Agents per GPU | Unsolved | Out of time, still working | Out of time, looping | Finished wrong | Never started | Looked up the answer | Other error |
|---|---|---|---|---|---|---|---|---|
| terminus-2 | 8 | 17 | 14 | 0 | 3 | 0 | 0 | 0 |
| terminus-2 | 16 | 60 | 54 | 1 | 5 | 0 | 0 | 0 |
| terminus-2 | 32 | 127 | 116 | 1 | 9 | 1 | 0 | 0 |
| terminus-2 | 48 | 48 | 46 | 1 | 1 | 0 | 0 | 0 |
| terminus-2 | 64 | 56 | 54 | 1 | 1 | 0 | 0 | 0 |
| mini-swe-agent | 8 | 18 | 13 | 0 | 3 | 2 | 0 | 0 |
| mini-swe-agent | 16 | 61 | 51 | 0 | 3 | 6 | 0 | 1 |
| mini-swe-agent | 32 | 101 | 92 | 0 | 1 | 8 | 0 | 0 |
| mini-swe-agent | 48 | 47 | 43 | 0 | 0 | 4 | 0 | 0 |
| mini-swe-agent | 64 | 53 | 48 | 0 | 1 | 4 | 0 | 0 |
Out of time, still working: Hit the task's agent time limit while its last 8 actions were still mostly new (fewer than 50% repeat an earlier action). Out of time, looping: Hit the time limit while at least 50% of its last 8 actions repeated earlier actions of the same trial. Finished wrong: The agent stopped on its own and the task's tests (or GAIA's exact-match check) failed. Never started: The harness never ran (e.g. its in-sandbox install failed); counted unsolved. Looked up the answer: A tool call referenced the benchmark itself (its repo or site), e.g. to copy an expected answer; never counted solved even if the verifier passed. Other error: Any other exception (crashes, API errors, context overflow).
Pre-registered testsPre-registered tests1 table
Before the final runs we wrote down four claims and how we would test them, so the exploratory results above could be checked on fresh data.
Pre-registered tests
Four claims written down before the confirming runs; only runs recorded afterwards count. Effects in whole points, Holm-corrected for C1 to C3.
| Claim | Effect (points, 95% CI) | Verdict |
|---|---|---|
| C1: H200 vs H100 at 32 agents helps mini-swe-agent more than terminus-2 | +9 [−7, 24] | not confirmed |
| C2: 32 vs 16 agents on H100 hurts mini-swe-agent more | −9 [−24, 5] | not confirmed |
| C3: thinking off interacts with the harness | −3 [−17, 10] | not confirmed |
| C4: time-out share rises with median seconds per agent step | correlation | confirmed |
How we ran it
- Model and server
- Qwen3.8-27B (dense, BF16), Qwen3.8-27B (official FP8 checkpoint), Qwen3.6-35B-A3B (MoE, 3B active, FP8), served by us with vLLM 0.31.0. A condition is harness x serving config x number of concurrent agents on one GPU server. Configs held constant: 64K context, Qwen reasoning and tool parsers, --language-model-only, --enable-prompt-tokens-details, --max-num-seqs 128. Every run starts a fresh server (no warm prefix cache).
- Agents
- Four harnesses run unchanged: terminus-2 (Harbor's own agent, driving a tmux session), mini-swe-agent (bash only, actions parsed from text), opencode and pi (native tool calls). Each harness's own tools inside a sandbox per task (shell/tmux; opencode and pi native tool calls). Sandboxes have internet access.
- Tasks
- Terminal-Bench: random sample (seed 0) of tasks with an agent time limit of 20 minutes or less (63% of the benchmark), to fit the budget; GAIA: 60 of 165 tasks stratified by level (seed 0), 10-minute agent limit. Task lists fixed before the first run. Terminal-Bench: 36 of 89 tasks; GAIA: 60 of 165.
- Limits
- Terminal-Bench agents get each task's own time limit (600 to 1,200 seconds; 900 seconds for 32 of the 36 tasks), doubled or quadrupled in the extra-time settings; GAIA agents get 600 seconds. 65,536-token context, 16,384 output tokens per call.
- Grading
- Terminal-Bench task tests: solved if verifier reward >= 1.0; GAIA answer check: solved if verifier reward >= 1.0. No model grades itself. Trials whose tool calls reference the benchmark are never counted solved, even when the verifier passed.
- Noise
- Solve-rate intervals resample tasks with all their trials (2,000 resamples); differences are paired by task. Identical conditions differed by up to 16.67 points between runs, and 17-20% of same-task rerun pairs disagreed on pass or fail. A difference whose interval includes zero is treated as no difference.
- Cost
- GPU cost only: run time while the agents ran × list price per GPU-hour (H100 $3.95, H200 $4.54, B200 $6.25, October 2026), divided by solved tasks. Sandbox cost is left out because we only have an estimate of it, and the estimate under-counts the bill.
- Excluded runs
- 5 runs produced no trials at all (server deploy failures, killed processes, a bad task list) and are excluded. Every attempted trial is counted, including ones that crashed, timed out or never started.
What it can't show
- Two open models (Qwen3.8-27B dense, Qwen3.6-35B-A3B MoE) and one serving engine (vLLM 0.31.0); cloud GPUs and prices as of October 2026.
- The 36 Terminal-Bench tasks all have agent limits of 20 minutes or less (63% of the benchmark does), so effects that work through the time limit are larger than on the full benchmark.
- Several cells rest on one run of 36 tasks; identical conditions differed by up to 16.67 points between runs. Claims rest on paired intervals and on cells with 108+ trials.
- Harness x infrastructure interactions are exploratory; the three pre-registered interaction claims (C1-C3) were not confirmed on fresh data. Only C4 (time-out share follows server speed) was confirmed.
- Many paired comparisons were made; some intervals excluding zero are expected by chance.
- Benchmark-lookup detection is a regex over tool calls and misses lookups that never name the benchmark. Sandboxes had internet access.
- In-sandbox harnesses (mini-swe-agent, opencode, pi) never start on one task (qemu-startup, a Debian mirror 404), counted unsolved in every config; this lowers their absolute scores but not within-harness comparisons.
- Sandbox cost is an estimate from requested CPU/memory and sandbox lifetime; it under-counts the bill (in-flight, aborted and orphaned sandboxes).
- Run-to-run time varies about 20% for identical configs (stragglers), so per-cell cost per solve is noisy; GPU price is the October 2026 list rate.
- GAIA: 60 tasks, three cells per harness, one run each; no uncapped (extra-time) run.
- terminus-2 drove its agents from the session container through an egress proxy; the other harnesses called the server from sandboxes.
- The quickcheck (8 tasks) and the opencode/pi cells (36 trials each) are too small for firm claims.
The data
GAIA
Web research questions with exact-match answers. Gated on Hugging Face: accept its terms there to browse it. We used 60 of its 165 tasks, stratified by level.
Further reading
Run it on
your own agent
Kitaru turns your agent's traces into test cases and replays them against any model, prompt or setting, so you can see what a change does before it ships.