Why 77.4% Is Not Enough
On AppWorld’s test_normal evaluation, a ReAct agent backed by GPT-4.1 achieved a 77.4% average success rate over five runs. That sounds like evidence that the system completes most tasks, but it leaves out the question that matters more in production: will the outcome remain the same when the user submits the same request again?
When each task was repeated five times, only 53.0% succeeded on every run. The 24.4-point gap between average success and repeated full success reached 30 points on hard tasks. Nearly a quarter of the tasks were therefore neither simply solvable nor unsolvable; they were unstable. Leaderboards tend to foreground Mean@5, while Pass⁵ captures the stricter production requirement that every attempt must succeed.
Instability Lives in Small Decisions
A ReAct agent does not produce an answer in one step. It repeatedly decides what to do next: which API to call, which arguments to provide, whether to retry, and how to interpret a tool response. Each decision is drawn from a probability distribution over next tokens. If one option is clearly ahead, small computational perturbations rarely change it; when several options are nearly tied, floating-point effects, request batching, and similar serving details can send execution down a different path.
That is why temperature zero or a fixed random seed does not automatically create production-grade determinism. The weakness may be a local decision rather than a lack of overall model capability. A different API argument or a different interpretation of a search result can compound across tool calls and turn an otherwise successful workflow into a failure. Reliability is therefore a property of the execution chain, not just of the base model.
Turning One Trace into a Reliability Repair
The Consistency Analyzer does not repeatedly rerun the entire task. It starts with a recorded trajectory and looks for decision points that are prone to flipping. At each point, it requests multiple completions, with five as the default, and checks where the agent is likely to choose different actions. The procedure needs neither ground-truth answers nor access to the model’s internal state, making it an offline scan for fragile parts of an execution path.
ALTK-Evolve then turns those findings into reusable consistency guidelines and injects them during inference on similar tasks. In the AppWorld example, ambiguity around how to count items was narrowed by rules that anchored the match to whole lines and required checking multiple search results. The important move is not making the model generally smarter, but converting an observed fragile choice into an operational constraint that can be reused.
The Results Improve, but the Risk Remains
With consistency guidelines enabled, Pass⁵ rose from 53.0% to 69.0%, narrowing the consistency gap from 24.4 to 12.0 percentage points. Mean@5 also increased from 77.4% to 81.0%, so the repair did not trade average accuracy for stability. Pass⁵ on similar tasks improved by 13.0 points, suggesting that the guidelines were not useful only for the exact task that produced them.
This is still not evidence that the agent has become reliable in general. The analyzer depends on a recorded trajectory: it can expose unstable points on that path, but it may miss alternatives that were never taken. It also requires additional offline model calls, and the reported validation is centered on AppWorld and the evaluated model configurations. For workflows such as financial reconciliation or contract-obligation checks, the practical rule is to report Mean@k and Pass^k together, inspect the decision points that flip, and add verification, recovery, or human approval where retries are unsafe.