Does the serving engine change what one GPU serves?
We replayed the same recorded coding-agent traffic against stock vLLM and SGLang, and two tuned SGLang settings, serving Qwen3.8-27B on one H100. One run per setting: this is what happened for this model and this traffic, not a general verdict on either engine.
- Status
- Complete
- Run
- October 9, 2026
- Model
- Qwen3.8-27B (open weights)
- Hardware
- One H100
- Traffic
- 124 sessions, 1,465 turns
- Settings
- 4, one run each
The result
Requests completed on the same traffic, one H100 · higher is better - Stock vLLM
- 504
- Stock SGLang
- 277
- SGLang, radix cache off
- 435
Replayed coding-agent traffic. The third bar is SGLang's best setting.
- 01
Stock SGLang completed 277 requests to stock vLLM's 504 on the same traffic and GPU, and generated 205.79 tokens per second to vLLM's 391.94 (0.53×).
- 02
Stock SGLang ran at most 5 requests at once for this hybrid-attention model, against vLLM's 31, with up to 27 waiting.
- 03
Once a request ran, stock SGLang generated faster: a median 22.51 ms per output token against vLLM's 53.94 ms. The time went to waiting: median time to first token was 76.29 s against 8.66 s, and median request time 88.09 s against 27.92 s.
Results
Stock vLLM is highlighted as the reference. Read throughput and concurrency together: the engines differ less in generation speed than in how many requests they run at once.
Engines and settings
Every setting ran the same traffic on one H100, 32 concurrent replayed sessions, a 3 s gap between turns and a 720 s replay window.
| Setting | Engine | Extra flags |
|---|---|---|
| stock vLLM | vLLM 0.31.0 | none (the agent study's defaults) |
| stock SGLang | SGLang 0.5.20 | none (equivalent common flags) |
| SGLang, lazy radix + bf16 states | SGLang 0.5.20 | --mem-fraction-static 0.85, --mamba-radix-cache-strategy extra_buffer_lazy, --mamba-ssm-dtype bfloat16, --max-mamba-cache-size 128 |
| SGLang, radix cache off | SGLang 0.5.20 | --mem-fraction-static 0.85, --disable-radix-cache, --max-mamba-cache-size 48 |
What else we found
- 04
The best SGLang setting we tried, radix cache off, reached 0.86× stock vLLM's completed requests and 0.84× its tokens per second, with 11 requests reset by the server.
- 05
Each setting ran once, on one model, one GPU type and one workload, and we did not measure run-to-run variation. Read these as what happened here, not as a general ranking of the engines.
Full results
Every table behind the findings, grouped by question. Each opens in place.
More resultsThroughput · Latency · Concurrency and memory3 tables
Throughput
Requests and tokens served in one run per setting. Throughput divides by the measured window, which runs past 720 s while in-flight requests finish.
| Setting | Requests sent | Completed | Errored | Sessions reached (of 124) | Measured window | Generated tokens per second | Completed vs. stock vLLM | Tokens per second vs. stock vLLM |
|---|---|---|---|---|---|---|---|---|
| stock vLLM | 504 | 504 | 0 | 55 | 832.8 s | 391.94 | 1.00× | 1.00× |
| stock SGLang | 277 | 277 | 0 | 39 | 815.2 s | 205.79 | 0.55× | 0.53× |
| SGLang, lazy radix + bf16 states | 417 | 417 | 0 | 47 | 903.2 s | 301.97 | 0.83× | 0.77× |
| SGLang, radix cache off | 446 | 435 | 11 | 47 | 855.4 s | 328.68 | 0.86× | 0.84× |
Latency
Over completed requests. Time to first token includes time spent queued; time per output token is the generation speed once a request runs.
| Setting | Median request | 90th percentile request | Median time to first token | Median time per output token |
|---|---|---|---|---|
| stock vLLM | 27.92 s | 107.44 s | 8.66 s | 53.94 ms |
| stock SGLang | 88.09 s | 142.25 s | 76.29 s | 22.51 ms |
| SGLang, lazy radix + bf16 states | 54.04 s | 115.88 s | 36.49 s | 31.34 ms |
| SGLang, radix cache off | 45.97 s | 95.75 s | 29.39 s | 38.52 ms |
Concurrency and memory
Engine gauges scraped every 10 s. 32 sessions were always asking; requests the engine could not run waited in its queue.
| Setting | Peak running requests | Peak waiting requests | Peak cache usage (each engine's own gauge) | Prefix cache hit (client side) | Server startup |
|---|---|---|---|---|---|
| stock vLLM | 31 | 11 | 100.0% | 6.5% | 163.6 s |
| stock SGLang | 5 | 27 | 76.0% | 8.3% | 167.2 s |
| SGLang, lazy radix + bf16 states | 27 | 26 | 98.0% | 9.9% | 167.6 s |
| SGLang, radix cache off | 31 | 24 | 99.0% | 0.0% | 210.9 s |
ErrorsErrors1 table
Errors
Errored requests are left out of throughput and latency. Only one setting had any.
| Setting | Errored requests | Error |
|---|---|---|
| stock vLLM | 0 | none |
| stock SGLang | 0 | none |
| SGLang, lazy radix + bf16 states | 0 | none |
| SGLang, radix cache off | 11 | ConnectionResetError |
ConnectionResetError: Request failed with a connection reset from the server. A streamed request was reset by the SGLang server while the radix cache was off.
How we ran it
- Traffic
- Recorded mini-swe-agent sessions from our own Terminal-Bench 2.0 runs at 32 agents with vLLM defaults: 124 sessions and 1,465 model turns. Turns whose recorded prompt exceeded 60,000 tokens were cut. The recordings are not published.
- Replay
- No agent runs and no tools execute. None executed: each request re-sends the conversation exactly as the agent sent it before its i-th reply, with the bash tool schema, and asks for exactly as many tokens as the recorded reply (ignore_eos, temperature 0.6, streaming). Each run replays sessions in a fixed order on one H100, 32 concurrent replayed sessions, a 3 s gap between turns and a 720 s replay window, so runs reach different numbers of sessions.
- Engines
- vLLM 0.31.0 with the agent study's default flags, and SGLang 0.5.20 with the equivalent common flags, plus two non-default SGLang settings from its cookbook. Model: Qwen3.8-27B (dense, hybrid Gated DeltaNet + attention), open weights, self-hosted.
- requests completed
- Requests that returned without error and with a usage block within the run.
- generated tokens per s
- Completion tokens of completed requests / measured_window_s (the actual replay duration, which runs past 720 s while in-flight requests finish).
- request latency p50 s / p90 s
- Median / 90th percentile of request wall time over completed requests.
- ttft p50 s
- Median time to the first streamed chunk.
- time per output token p50 ms
- Median of (latency - ttft) / (completion tokens - 1).
- running requests peak
- Peak of the engine's running-requests gauge, scraped every 10 s.
- Noise
- One run per setting, with no repeats, so there is no run-to-run noise estimate and no confidence intervals.
What it can't show
- One run per setting; no repeats or confidence intervals.
- One model (a hybrid Gated DeltaNet + attention model, where SGLang's per-request linear-attention state is the bottleneck), one GPU type (H100), one workload (mini-swe-agent traffic at 32 sessions).
- Only two non-default SGLang settings were tried, following its cookbook; not an exhaustive tuning of either engine.
- Measures serving throughput and latency only, not agent outcomes.
- Replay re-sends recorded prompts with fixed generation lengths (ignore_eos); real agents would react to slower responses (fewer turns per time limit).
- Runs reached different numbers of sessions (39-55 of 124) within the window, so later sessions with longer prompts are weighted differently across engines.
- Replayed prompts were a median 0.8769-0.9256 of the recorded prompt token counts (reasoning text is not re-sent), so prefix sharing is close to but not identical with the original run.
Further reading
Run it on
your own agent
Kitaru turns your agent's traces into test cases and replays them against any model, prompt or setting, so you can see what a change does before it ships.