Can a self-hosted agent run ML training operations?
A self-hosted 27B model fixes injected faults in training jobs: distributed runs, checkpoints, out-of-memory errors, stragglers and scheduling quotas. We measured how GPU, load and serving settings change how many it solves. The study is stopped.
- Status
- Stopped: frontier agents saturated the tasks
- Run
- October 10, 2026 to October 13, 2026
- Tasks
- 24, built for this study
- Model
- Qwen3.8-27B, self-hosted on vLLM
- Harnesses
- terminus-2, mini-swe-agent
- Grading
- Deterministic test per task
The result
Hard-set solve rate · higher is better - H100
- 34 / 48
- H200
- 45 / 48
terminus-2 with 24 agents on one GPU, 8 tasks.
- 01
We stopped building harder tasks: across five rounds, frontier coding agents on vendor APIs solved nearly every task, so the set no longer tells strong agents apart. Their results are not published here. What stands is the self-hosted result below.
- 02
On the 8-task hard set, an H200 instead of an H100 at 24 agents raised Qwen3.8-27B's solve rate by +22.9 points with terminus-2 and +18.8 with mini-swe-agent, both with intervals above zero.
- 03
On the harder v2 tasks the GPU mattered more: terminus-2 solved 2 of 24 trials on the H100 and 11 of 24 on the H200.
Results
On the 6-task pilot every setup solved 30 of 30 trials, so the pilot showed no serving effect. The tables below are the hard and v2 sets, where it did.
Hard set: what moves the solve rate
Qwen3.8-27B on 8 training-operations tasks, 3 attempts per task per run, each setup compared with the same model on one H100 serving 24 agents at once.
| Setup | terminus-2 solved | vs. baseline (points) | mini-swe-agent solved | vs. baseline (points) |
|---|---|---|---|---|
| H100, 24 agents (baseline) | 34 / 48 | baseline | 31 / 48 | baseline |
| 8 agents instead of 24 | 40 / 48 | +12.5 [−4.2, 31.3] | 43 / 48 | +25.0 [8.3, 41.7] |
| H200 instead of H100 | 45 / 48 | +22.9 [4.2, 43.8] | 40 / 48 | +18.8 [6.3, 31.3] |
| Time limit doubled to 60 min | 46 / 48 | +25.0 [10.4, 41.7] | 34 / 48 | +6.3 [−2.1, 16.7] |
| Output capped at 4,096 tokens | 20 / 24 * | +12.5 [2.1, 27.1] | 16 / 24 * | +2.1 [−18.8, 20.8] |
| Output capped at 8,192 tokens | 43 / 48 | +18.8 [0.0, 37.5] | 38 / 48 | +14.6 [2.1, 27.1] |
| Output capped at 16,384 tokens | 19 / 24 * | +8.3 [−2.1, 20.8] | 19 / 24 * | +14.6 [−2.1, 31.3] |
| Reasoning off | 30 / 48 | −8.3 [−22.9, 6.3] | 23 / 48 | −16.7 [−37.5, 2.1] |
| MoE model (Qwen3.6-35B-A3B) | 14 / 24 * | −12.5 [−45.8, 16.7] | 11 / 24 * | −18.8 [−43.8, 6.3] |
* One run of 3 attempts per task (24 trials); every other row pools two runs (48 trials). Differences are paired against the baseline over the same 8 tasks; a 95% interval that includes 0 is no difference. Highlighted: setups that helped with both harnesses beyond noise.
What else we found
- 04
Fewer agents per GPU helped mini-swe-agent: 8 instead of 24 raised its solve rate by +25.0 points [8.3, 41.7], while terminus-2's +12.5 [−4.2, 31.3] was within noise. Capping output at 8,192 tokens per call raised it by +14.6 [2.1, 27.1], and gave terminus-2 its lowest GPU cost per solve on an H100: $0.0960 against $0.1229 at defaults.
- 05
Most unsolved trials ran out of time: 144 of Qwen3.8-27B's 217 unsolved trials hit the time limit. Identical runs differed by up to 7 solves out of 24, so single-run differences are rough.
Full results
Every table behind the findings, grouped by question. Each opens in place.
More resultsHarder v2 tasks: H100 vs. H200 · GPU cost per solved task2 tables
Harder v2 tasks: H100 vs. H200
4 tasks (3 new plus one from the hard set), 6 attempts per task, one run per GPU, 24 agents at once.
| Harness | H100 solved | H200 solved | Difference (points, 95% CI) |
|---|---|---|---|
| terminus-2 | 2 / 24 | 11 / 24 | +37.5 [8.3, 66.7] |
| mini-swe-agent | 4 / 24 | 13 / 24 | +37.5 [16.7, 58.3] |
Paired over 4 tasks, so the intervals are wide.
GPU cost per solved task
Hard set. GPU time for the whole run at the list hourly price, divided by solved trials. Sandbox cost is left out (it was estimated, not billed per run).
| Setup | GPU | terminus-2 | mini-swe-agent |
|---|---|---|---|
| H100, 24 agents (baseline) | H100 | $0.1229 | $0.1383 |
| 8 agents instead of 24 | H100 | $0.2079 | $0.1800 |
| H200 instead of H100 | H200 | $0.1052 | $0.1164 |
| Time limit doubled to 60 min | H100 | $0.1752 | $0.2400 |
| Output capped at 4,096 tokens | H100 | $0.1027 | $0.0980 |
| Output capped at 8,192 tokens | H100 | $0.0960 | $0.1202 |
| Output capped at 16,384 tokens | H100 | $0.1093 | $0.1077 |
| Reasoning off | H100 | $0.1362 | $0.1859 |
| MoE model (Qwen3.6-35B-A3B) | H100 | $0.1441 | $0.1841 |
By task familyBy task family, terminus-2 · By task family, mini-swe-agent2 tables
The hard set has two tasks in each of four families: distributed jobs, diagnosis from telemetry, checkpointing and recovery, and resource limits and scheduling.
By task family, terminus-2
Hard set, 2 tasks per family. Solved trials per family and setup.
| Setup | A. distributed jobs | B. diagnosis from telemetry | C. checkpointing and recovery | D. resource limits and scheduling |
|---|---|---|---|---|
| H100, 24 agents (baseline) | 8 / 12 | 10 / 12 | 6 / 12 | 10 / 12 |
| 8 agents instead of 24 | 12 / 12 | 11 / 12 | 6 / 12 | 11 / 12 |
| H200 instead of H100 | 12 / 12 | 11 / 12 | 11 / 12 | 11 / 12 |
| Time limit doubled to 60 min | 11 / 12 | 12 / 12 | 11 / 12 | 12 / 12 |
| Output capped at 4,096 tokens | 4 / 6 | 6 / 6 | 5 / 6 | 5 / 6 |
| Output capped at 8,192 tokens | 11 / 12 | 10 / 12 | 11 / 12 | 11 / 12 |
| Output capped at 16,384 tokens | 5 / 6 | 6 / 6 | 3 / 6 | 5 / 6 |
| Reasoning off | 6 / 12 | 11 / 12 | 5 / 12 | 8 / 12 |
| MoE model (Qwen3.6-35B-A3B) | 4 / 6 | 3 / 6 | 4 / 6 | 3 / 6 |
Darker cells missed more trials in that family.
By task family, mini-swe-agent
Hard set, 2 tasks per family. Solved trials per family and setup.
| Setup | A. distributed jobs | B. diagnosis from telemetry | C. checkpointing and recovery | D. resource limits and scheduling |
|---|---|---|---|---|
| H100, 24 agents (baseline) | 8 / 12 | 11 / 12 | 4 / 12 | 8 / 12 |
| 8 agents instead of 24 | 9 / 12 | 12 / 12 | 10 / 12 | 12 / 12 |
| H200 instead of H100 | 12 / 12 | 10 / 12 | 7 / 12 | 11 / 12 |
| Time limit doubled to 60 min | 7 / 12 | 12 / 12 | 6 / 12 | 9 / 12 |
| Output capped at 4,096 tokens | 3 / 6 | 5 / 6 | 3 / 6 | 5 / 6 |
| Output capped at 8,192 tokens | 9 / 12 | 10 / 12 | 8 / 12 | 11 / 12 |
| Output capped at 16,384 tokens | 5 / 6 | 5 / 6 | 3 / 6 | 6 / 6 |
| Reasoning off | 5 / 12 | 6 / 12 | 5 / 12 | 7 / 12 |
| MoE model (Qwen3.6-35B-A3B) | 3 / 6 | 3 / 6 | 3 / 6 | 2 / 6 |
Darker cells missed more trials in that family.
FailuresHow the unsolved trials ended1 table
How the unsolved trials ended
Every self-hosted run, all task sets. Unsolved trials by how they ended.
| Model | Harness exited with an error | Finished, test failed | Sandbox ran out of memory | Hit the time limit |
|---|---|---|---|---|
| Qwen3.8-27B (dense) | 16 | 56 | 1 | 144 |
| Qwen3.6-35B-A3B (MoE, FP8) | 2 | 18 | 0 | 3 |
How we ran it
- Tasks
- Tasks built for this benchmark: injected faults in multi-process training setups, each graded by an independent test (Harbor task format). Families from published failure taxonomies (ByteRobust, L4, SIGMA). 24 tasks in five rounds: a 6-task pilot, an 8-task hard set (families A to D), a 4-task v2 set, 5 v3 tasks and 2 tasks on a real training stack (torchtitan, 2 GPUs). The self-hosted model ran the pilot, hard and v2 sets.
- Agent
- Qwen3.8-27B (and, once per harness, Qwen3.6-35B-A3B in FP8) served by vLLM 0.31.0 on our own GPUs with a 65,536-token context. Two open harnesses drive it: terminus-2 (a tmux session) and mini-swe-agent (bash). Each trial runs in its own sandbox.
- Limits
- 30 minutes per trial on the hard and v2 sets (20 on the pilot; 60 in the doubled-time setup).
- Grading
- A deterministic test per task recomputes the result, injects faults and checks hidden seeds (and golden loss curves on the torchtitan tasks). No model grades. Every task was checked both ways: the reference fix passes and doing nothing fails; three hard-set tasks were also checked with a partial fix that fails. Timeouts, harness errors and setup failures count as unsolved.
- Noise
- Every self-hosted hard-set setup ran twice, 3 attempts per task each, except the 4,096 and 16,384-token caps and the MoE model. Identical runs differed by up to 7 solves out of 24 (mini-swe-agent with reasoning off: 8 and 15); the H100 baseline repeated at 17 and 17 (terminus-2) and 14 and 17 (mini-swe-agent). Differences are paired over tasks with a bootstrap (10,000 resamples); a 95% interval that includes 0 is no difference.
- Cost
- GPU cost per solved task = GPU time for the whole run at the list hourly price (H100 or H200), divided by solved trials. Sandbox cost is not included.
What it can't show
- Small task sets: 6 (v1), 8 (hard), 4 (v2), 5 (v3), 2 (torchtitan); bootstrap CIs over 4-8 tasks are wide.
- Tasks were built by the same team that ran the agents, and later rounds were designed after seeing earlier results.
- One self-hosted model family (Qwen3.8-27B, plus one MoE run per harness) on one engine (vLLM 0.31.0).
- The 4k/16k output caps and the MoE model are single runs.
- Self-hosted failure categories come from Harbor exception types; they do not separate looping from working timeouts.
- The v3 and torchtitan tasks were only run with frontier agents on vendor APIs; those results are not published here.
- Cost per solve counts GPU time only; the sandbox cost was estimated and is left out.
Run it on
your own agent
Kitaru turns your agent's traces into test cases and replays them against any model, prompt or setting, so you can see what a change does before it ships.