How does a basic agent with GPT-4.1 beat fine-tuned models on a customer-service benchmark?

What τ²-bench actually measures

τ²-bench comes from Sierra. It evaluates how well an agent handles customer-service conversations under real constraints. Each task pairs the agent with an LLM-simulated customer who has a goal, a set of domain tools, and a written policy document the agent must follow. The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions. A task counts as solved only when the final database state matches the expected outcome, a strict binary reward. The benchmark covers two domains: retail and airline.

IBM runs the benchmark's own implementation of a basic ReAct agent with a two-phase THINK → ACT loop. GPT-4.1 serves as the agent model. GPT-5.5 serves as the customer simulator. Four trials per task. No weight updates needed.

How the guidelines are built and applied

ALTK-Evolve analyzes the agent's training-task attempts. It uses the same model — GPT-4.1 — to distill successes and failures into short behavioral rules. Those guidelines are then appended to the agent's system prompt as a plain markdown list after the domain policy. No weight updates or human annotation are needed.

Each guideline carries a support count: the number of distinct training tasks it was extracted from. Higher support means a rule that generalized across more tasks rather than a one-off. Three configurations are compared: baseline with no guidelines, sup3 keeping only guidelines with support ≥ 3, and lossless keeping every guideline retained after consolidation (support ≥ 1). On retail the lossless set contains 292 guidelines; sup3 holds 31. On airline lossless has 101; sup3 has 14.

What the numbers show on retail and airline

Every task is attempted four times. Mean Accuracy is the average success rate across the four trials. Pass^4 is the percentage of tasks that passed across all four trials — the metric the τ²-bench leaderboard reports.

On retail with GPT-4.1, baseline Mean Accuracy sits at 78.1% and Pass^4 at 57.5%. The sup3 core barely moves Mean Accuracy to 78.8% but lifts Pass^4 to 60.0%. The full lossless set pushes Mean Accuracy to 82.5% and Pass^4 to 67.5%.

On airline with GPT-4.1, baseline Mean Accuracy is 57.5% and Pass^4 35.0%. The sup3 core raises Mean Accuracy to 60.0% and Pass^4 to 45.0%. The lossless set raises Mean Accuracy further to 65.0% but Pass^4 drops back to 40.0%.

The two domains disagree about how much guidance pays off. In retail more is better on every metric. In airline the compact sup3 core is the more reliable one at the reliability end.

Cost and latency trade-offs

Guidelines aren't free. The cost lands in prompt input tokens, not in longer conversations.

Retail baseline conversations cost $0.084. Sup3 adds 12% to reach $0.095. Lossless adds 65% to reach $0.139. Airline baseline is $0.081. Sup3 adds 19% to $0.097. Lossless adds 32% to $0.107.

To check whether the agent talks longer, IBM compared matched runs solved under both baseline and guideline variants — identical scenario and seed. On retail the shared set of 111 tasks shows baseline turns at 10.59 and lossless at 10.16, a reduction of 0.43 turns. On airline 40 shared tasks show baseline at 9.15 turns and lossless at 8.82, a reduction of 0.33 turns. The agent isn't talking longer. Guidelines make conversations slightly shorter. Each turn is a sequential LLM round-trip. Fewer turns means lower end-to-end latency. In production much of the input-cost increase is reclaimable with prompt caching since the guideline block is identical across every call.

Works with stronger models too

A fair objection: learned guidelines might only patch gaps of a weaker model. IBM tested this directly by rerunning airline using GPT-5.5 for both the agent and guideline extraction on the same ReAct scaffold. None of the extracted guidelines reached support ≥ 3; they top out at support 2, so the high-support core becomes sup2 rather than sup3.

With GPT-5.5 on the same 20 held-out airline tasks, baseline Accuracy is 72.5% and Pass^4 55.0%. Sup2 with 103 guidelines lifts Accuracy to 80.0% and Pass^4 to 70.0%. Lossless with 146 guidelines lifts Accuracy to 82.5% and Pass^4 to 70.0%. The reliability gain of +15 points is larger than the +10 points observed with GPT-4.1 on these very tasks, and from a much higher floor. Where the baseline's reliability swung between reruns, the lossless configuration reproduced exactly.

Leaderboard position and what it means

Submissions are scored on the full base split (train ∪ test) as τ²-bench requires, reported at Pass^4. On retail the agent with learned guidelines reaches Pass^4 = 67.5%, ahead of every frontier model and ahead of RAFT-30B-A3B at 61.4%, a model fine-tuned on synthetic τ²-bench retail data. Using stock GPT-4.1 and a prompt-appended playbook, it outperforms that fine-tuned system by 6.1 points.

The airline compact sup3 core gives the GPT-4.1 agent Pass^4 = 44%, placing 8th, beating GPT-5.2 (no reasoning) and nearly matching Claude Sonnet 4.5. Swap in GPT-5.5 and the same approach tops the field outright at 82.0%, clearing every frontier entry in the 66–72% range. The leaderboard snapshot is from October 12, 2026.

Where the explanation stops

The document does not say how many tasks are in the full base split versus the held-out test split. It does not describe the consolidation process that produces the final guideline sets. It does not report results for other domains or benchmarks. It does not specify whether the guidelines are static after extraction or can be updated incrementally. It does not compare latency in wall-clock time, only in assistant turns. It does not disclose the compute or time required to mine guidelines from training trajectories.