Evidence at a glance
A Successful Tool Call Can Still Leave the Wrong Result
Microsoft and Hugging Face have released ThinkingBox-Training, which turns ThinkingBox's executable checks of agent outcomes into reinforcement-learning rewards. It targets workflows where an agent must operate a real backend, follow business rules, and leave the right records behind. The aim is not simply to train a model that sounds more helpful, but one that changes the system correctly after using its tools.
That distinction is not semantic. In an earlier ThinkingBox example, an agent made nine plausible tool calls and marked a customer ticket as resolved even though the carrier exception remained open. Nothing in the tool sequence necessarily looked broken, and the agent could still claim it had finished. The new step is to move this kind of terminal-state check from evaluation into the training loop, so what happened in the database helps shape the agent's behavior as well as determine its score.
Reward the Outcome, Not a Prescribed Path
ThinkingBox-Bench contains 507 tasks. Each starts a backend, gives the agent MCP tools, and uses a simulated user who may hold private context. When the agent finishes, the evaluator checks the final database state and relevant side effects rather than requiring a prescribed sequence of tool calls. If the business records meet the goal, different paths may count as success; a human-written procedure is not treated as the only valid solution.
For requirements that can be checked directly in state, the evaluator returns a pass or fail that can become a binary reward. Of the benchmark tasks, 477 are scored on state alone, while 30 also use a response rubric for requirements that cannot be represented directly in database fields, such as whether the agent made clear that an arrangement was not guaranteed. This extends scoring to some aspects of communication, but it also means the grader is not purely mechanical and reproducible in every case. The evaluator's limits become limits on the training objective.
The Gain Is Consistency, Not Broader Coverage
Microsoft and Hugging Face built a training pool from ThinkingBox-format tasks outside the benchmark. They first estimated each task's success probability under the base policy, then selected tasks that were neither too easy nor hopeless for that model. Difficulty is policy-relative: once the model learns a task, that task is no longer equally difficult. In this experiment, however, the pool was selected once before training and held fixed through 50 GRPO steps.
For Qwen3.8-27B, pass@1 on ThinkingBox-Bench rose from 51.70% to 60.78% after training. More strikingly, the number of tasks passed on all 20 attempts increased from 38 to 107, while the share passed at least once slipped from 89.35% to 87.97%. Here, pass@20 means that a task succeeded at least once in 20 runs; “20/20” means it succeeded in every run. The results show a specific trade-off: the model became more consistent on a set of tasks, but its coverage did not grow with that consistency.
A Hotel Workflow Shows the Behavioral Shift
A corporate hotel-booking task makes the behavioral shift concrete. A traveler requests a room downgrade, but company policy requires manager approval first. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After training, it preserved the booking and payment state in all 20 attempts, and all expected records matched in 19.
This is not merely a lesson to avoid one specific action. The reward checks whether the agent leaves the outcome required by the business rule: when approval is missing, do not change the booking or payment prematurely. The example also shows why counting successful tool calls is insufficient. A refund call may execute perfectly and still be the side effect the system should prevent. If reward tracks only whether the call completed, it can encourage the wrong behavior.
Training Inherits the Grader's Blind Spots
ThinkingBox addresses a recurring failure pattern. A study covering 121,680 valid trials across 12 models found that 79,853 failed executable checks. Of those failures, 67.24% still ended normally, had called a state-changing tool, and reported no final tool error. The checks found incorrect field values, extra side effects, and missing required side effects; these categories can overlap. In other words, normal-looking logs do not establish correct business state. Terminal-state verification targets the gap between a workflow that appears to succeed and one that actually does.
But a better evaluator does not by itself prove reliability in deployment. ThinkingBox tasks are reconstructed synthetic workflows, and the published material does not establish that they represent unseen enterprise systems, permission combinations, or exceptional cases. A fixed pool also ages: as a model improves, tasks that once offered useful practice may become too easy or stop covering new failure modes. For technical leaders, a sensible first use is to turn existing acceptance checks for rule-bound, state-verifiable workflows into rewards, while tracking coverage, repeatability, and side effects together. An increase in tasks passed 20 out of 20 is not evidence that an agent is universally reliable.