Which model holds up as a telecom support agent?

How we ran it

Task

tau2-bench Telecom (commit 05736c2 (2026-10-21)), its 40-task test split: a support agent helps a simulated customer fix a phone problem (MMS, mobile data or no service). The agent can look up accounts and bills, change the line and roaming, and message the customer; the simulated customer performs the device actions. Nothing was tuned on these tasks.

Agent and replays

Our agent runs inside tau2's own orchestrator, with tau2's telecom tools, and Kitaru records each episode as a session. The GPT-5.2 baseline was recorded live; every other arm is a Kitaru replay of those 40 sessions with the model or the prompt changed. A replay starts a fresh episode from the task, so it is a new full run, not a step check. tau2's own user simulator with gpt-4.1, unchanged across arms; at most 100 steps and 20 minutes per episode.

Prompt rule

One general rule appended to GPT-5.2's instructions: work through every applicable manual section, have the user re-check after each change, check the account side, and transfer only when every step has failed or the manual says to escalate. Written from the failure pattern in the public Exgentic recordings (premature transfers) and the tau2 manual's structure; it contains no task ids, scenarios, device states or answers. When it was written relative to the baseline run was not recorded.

Grading

tau2's own grader checks the end state of the phone and the account; no LLM judge is involved. An episode with no graded outcome counts as unsolved; none occurred.

Noise

GPT-5.2 replayed with nothing changed. Wilson score 95% intervals for single rates; paired differences against the baseline use a per-task bootstrap, 10,000 resamples. A difference whose interval includes zero is reported as not distinguishable.

Public recordings

The runs are Exgentic's, published on Hugging Face as agent-llm-traces-v2. We did not run these agents. The published shards hold 10,056 rows (the dataset card says 10,057), 1,844 of them tau2-bench Telecom across four agent harnesses. We kept the tool-calling harness only, 499 runs, so every model ran the same agent design with native tool calls. Nothing was dropped within it.

Public recordings: task

tau2-bench Telecom: a support agent helps a simulated customer fix a phone problem (MMS, mobile data or no service). The agent can look up accounts and bills, change the line and roaming, and message the customer; the simulated customer performs the device actions. The simulated customer's model and cost are not in the data.

Public recordings: scoring

A run counts as solved when the dataset's success field is true, which is the benchmark's own evaluation as recorded. 288 runs have an "unknown" status while success is set; they are scored by the success field. Two GPT-5.2 runs carry another model's run label and are counted under the model their calls name.

Public recordings: cost

Cost per solved task uses each run's agent cost as Exgentic recorded it, summed over solved and unsolved runs and divided by the solved ones. The prices behind it and its treatment of cached tokens are not documented, and the simulated customer is excluded.

Step check

160 of GPT-5.2's 1,048 replayable recorded calls, sampled deterministically, were resent exactly as recorded (the conversation up to that step and the recorded tool list) to GPT-5.2 and to seven newer models, and each reply was compared with the recorded one, exactly, with no LLM judge. The recorded tool definitions carry no argument schemas, so every model saw tool names and descriptions only.

Step check intervals

Wilson score 95% intervals for single rates. Paired differences against GPT-5.2's repeat use a cluster bootstrap over the 19 recorded runs the sampled calls came from, 10,000 resamples. A difference whose interval includes zero is reported as not distinguishable.