Why Does an AI Agent Say the Job Is Done When the Database Still Shows an Open Problem?

The agent said resolved. The database said hold.

A customer writes in about a $745 kitchen appliance stuck at a Nashville distribution center fifteen days past its estimated delivery date. The AI agent makes nine tool calls: it pulls the order, checks tracking, reads the refund policy twice, confirms no ticket exists, opens one, documents the timeline, and closes the ticket as resolved. Its final message asks whether there is anything else it can help with. Two things are wrong. The carrier exception is still open, so the required end state was hold, not solved. The customer never received an answer to what she actually asked. An AI grader checking tool calls would see nine well-formed ones. A grader checking whether the agent wrote to the database would see that too. The database is what disagrees.

A tool call is not an outcome

Final responses and valid tool calls are only proxies. An agent can sound correct while leaving the wrong value, changing the wrong record, or creating an extra side effect. Only the records it leaves behind settle the question. In a common-set ablation covering 121,680 valid trials across twelve LLM models, 79,853 attempts failed the executable checks. Of those failures, 67.24 percent still terminated cleanly, invoked a state-changing tool, and reported no final tool error. Executable checks nevertheless found wrong field values in 77.61 percent of them, unintended extra effects in 43.30 percent, and missing required effects in 25.36 percent. Those state-check findings overlap. A trajectory is a claim. Database state is the evidence. Repetition is the trust test.

One success is not reliability

An agent that processes a refund correctly once and mishandles it the next four times is not a working refund agent. So every task runs twenty independent times, each from an identical clean backend, and the benchmark reports three different numbers. Pass@1 is the share of all attempts that succeeded. Pass@20 is the share of tasks solved at least once in twenty tries. Observed 20/20 is the literal count of tasks that passed all twenty recorded attempts. No estimator, no smoothing.

The leaderboard shows Claude Opus 5.5 ahead overall at 67.16 percent pass@1, two-thirds of a point above Claude Opus 5. Kimi-K3 is the strongest open-weights model, within a point of GPT-6 Astra. Domain matters just as much: Claude Opus 4.6 scores 68.62 percent on retail but 8.30 percent on auto insurance. One good run tells you a model can do the work. It does not tell you whether it will do it again.

How much of that score survives twenty repeats

Only three models hold on to most of their pass@1 scores. GPT-6 Astra retains 78 percent of its single-attempt rate. Claude Opus 5.5 and Claude Opus 5 each retain 71 percent. At the other end, GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8 percent. The gap between what a model can do once and what it does every time is the whole story.

Breadth and consistency pull apart

Kimi-K3 has the broadest coverage of any model tested. It solves 93.89 percent of the benchmark at least once: 476 of 507 tasks. Only 31 tasks defeat it entirely, the lowest count in the field. On retail workflows it leads outright at 82.24 percent pass@1, ahead of every proprietary model. Kimi-K3 is also among the least consistent. Just 68 of 507 tasks, 13.41 percent, succeed in all twenty attempts.

The picture flips with Claude Opus 5. It solves fewer tasks at least once (79.09 percent; 106 defeat it entirely) but completes 47.53 percent of the benchmark on every single attempt. A newer model does not fix this. Claude Opus 5.5 scores higher than Claude Opus 5 on every-attempt average, 67.16 percent against 66.50 percent, and solves more tasks at least once. It passes exactly the same number of tasks on all twenty attempts: 241. Half a point of headline accuracy bought no additional dependability at all. Kimi-K3 solves 75 more tasks at least once than Opus 5. Opus 5 solves 173 more tasks consistently than Kimi-K3. If you are choosing a model for work that touches real records, pass@20 is the wrong column to look at.

What consistency costs

Capability comparisons usually stop at the score. For anyone deploying, the relevant question is what a successful unit of work costs. The benchmark measures that as cost per successful task attempt. Token usage from the full 507 × 20 campaign was priced at undiscounted list rates available on OpenRouter, reversing promotional discounts and excluding endpoints that declare quantization. Rates for input and output come from one provider endpoint per model. Cache rates use the same endpoint. Cost per successful task attempt equals estimated cost for 507 attempts divided by (507 × pass@1). This is a comparative efficiency index, not an invoice, and not the price of serving one production request. It also prices single successes, not consistency.

The Pareto cost frontier

A model is on the frontier if no other model is both no more expensive and at least as accurate. Three models qualify; every other model is dominated on at least one axis. GPT-5.6 Sol has the lowest cost per success at $0.127. GPT-5.4 raises pass@1 by 3.45 percentage points for $0.004 more per success. Claude Opus 5.5 adds another 1.80 points at $0.276 per success. Each of the three remains on the cost frontier line because no cheaper model matches its pass@1. Claude Opus 5 is the clearest case: at $0.475 per successful attempt and 66.50 percent pass@1, it is both more expensive and less accurate than Claude Opus 5.5 at $0.276 and 67.16 percent.

Now price consistency

Cost per success rewards a model that is cheap and often right. It does not reward a model that is right every time. So the benchmark also computes cost per dependable task: the cost of the full twenty-run campaign divided by the number of tasks the model passed on all twenty attempts. GPT-5.4 is the cheapest at $6.80, though only 128 tasks meet the bar. GPT-6 Astra reaches 231 at $7.45, and Claude Opus 5.5 the joint-highest 241 at $7.80. None of the three dominates the others: each additional dependable task costs more. Claude Opus 5 also passes 241, but at $13.30, so Opus 5.5 dominates it outright. GPT-5.6 Sol, the cheapest per single success at $0.127, costs $9.76 per dependable task. The cheapest way to get a right answer is not the cheapest way to get a dependable one.

Failure signatures

Each failed trace receives one deterministic diagnostic signature. The headline is actionable: roughly four in five failures are tool handling, not reasoning. Across an ablation study, tool usage accounts for 79.9 percent of failures. Wrong state updates account for 10.3 percent. Incomplete user resolutions account for 7.0 percent. No state-changing action accounts for 2.9 percent. These are unweighted averages of per-model shares and observable labels, not unique causal explanations. The practical pattern is simple: agents usually get far enough to attempt the workflow, then fail to recover from tool errors, failed preconditions, or empty lookups. That is a retry and error-recovery problem before it is a model problem. Difficulty also changes by domain: across the models listed above, retail averages 59.52 percent pass@1 while auto insurance averages 33.83 percent.

How the sandbox works

The sandbox is the agent environment, while ThinkingBox-Bench is a dataset benchmark to evaluate agents. Each task defines a starting backend state, a user goal, the available MCP tools, the domain policy, and executable checks over the terminal state. A simulated user holds private context — a booking reference, a preference, or a date of birth — and releases it only when asked. Every attempt gets an isolated MCP session with freshly initialized state. Two attempts of the same task never share a database row or cached tool state, which is what makes twenty-trial comparison meaningful. At the end, a side-effect extractor derives what actually changed, and deterministic judges compare it against the required end state, accepting any trajectory that produces the right outcome while rejecting wrong or missing effects. Extra effects are also rejected. For requirements with no clean database value, a narrow binary rubric question handles the semantics. 477 of the 507 tasks are graded on state alone; 30 add response rubrics. The trust boundary: the model sees tasks and dialogue. Tool schemas, golden state, assertions, grading internals and credentials stay on the evaluator side.

Run it yourself

The harness and the dataset are now on Hugging Face. ThinkingBox-Bench sits behind the OpenEnv interface, and each finished episode returns a binary pass/fail reward. The released adapter is designed for evaluation; separate, non-benchmark scenarios can use the same interface in training workflows. Tested on Linux and WSL, with Python 3.11+, uv and Docker. You also need a thinkingbox-data checkout at the pinned release and model endpoints for the agent, the simulated user, and the judge. One endpoint can serve all three roles, which is the simplest way to start.

Install OpenEnv and the ThinkingBox environment, then clone the executable benchmark at the pinned release thinkingbox-bench-v1.0, then install the ThinkingBox CLI which provides the tb command. Start Typesense 30.1 in a second terminal and wait for its health check. Start the Session Proxy and MCP servers in a third terminal. Back in the first terminal, start the OpenEnv server against a ThinkingBox YAML config naming your three models. Gate on readiness before running anything. It returns 503 until its observable data, configuration and Session Proxy checks pass. It cannot observe Typesense or live-probe every model endpoint, so confirm those separately. Score a real episode with the packaged evaluator. The