Evidence at a glance
One Image, Several Ways to Fail
In October 2026, Elorian introduced Glimpse, a visual reasoning model, and released DecoyCount with AI data company Mercor. The benchmark tests whether models can count objects in real-world images according to the intended rule. The task sounds simple: how many apples are in the picture? But if one apple appears in a mirror, a wrong answer could mean the model missed the real fruit—or counted its reflection as another apple.
Those failures call for different fixes. The first points toward visual perception; the second concerns how a model decides what belongs in the set being counted. A final number alone collapses both into the same error. DecoyCount is designed to expose that distinction, rather than merely pose a harder counting question.
Turning a Wrong Count into a Diagnosis
DecoyCount contains 1,660 real-world images and seven types of decoys: duplicates such as reflections, wrong counting units, prior expectations, lookalikes, text in the image, misleading structure, and estimation. These are not arbitrary distractions. Each represents a plausible source of error, such as treating a number printed on a package as the number of objects, or losing track of items because of how they are arranged.
Each image can be paired with different questions: a neutral one that does not state the rule, a matched prompt that clarifies the rule without giving away the answer, and a multistep question set in a practical scenario. The paper compares accuracy under neutral and matched prompts and calls the difference the “reasoning-alignment gap.” That gap indicates how much a model is thrown off by the counting rule; it is not a complete measure of either visual ability or reasoning.
A Better Prompt Helps, but Does Not Improve Vision by Itself
All six models reported in the paper scored below 50% on neutral questions. GPT-6 Astra led that group at 43.17%. Matched prompts raised accuracy by 11.77 to 22.09 percentage points: GPT-6 Astra gained 22.09 points, while Gemini-3.8-flash gained 11.77. Under neutral conditions, five of the six models selected a decoy answer more often than the true count. With matched prompts, decoy selections fell by 43.3% to 66.7%.
That pattern suggests that at least some failures are not simply a matter of failing to see an object. Models can also follow a misleading counting rule when the task leaves it implicit. But a prompt that changes the answer does not, by itself, show that the model sees better. Richer real-world scenarios did not consistently improve performance: they rescued only 11.6% to 32.6% of errors on neutral questions. Prompts helped less on estimation decoys, where perception or precise enumeration may matter more.
Read Glimpse’s Lead Alongside the Test Conditions
Latent Space reported Elorian CEO Andrew Dai’s claim that Glimpse scored 50.48% on the 1,660 DecoyCount items without cues. The same report listed GPT-6 Astra at 43.17%, Gemini-3.8-flash at 32.71%, Qwen3.8-max at 28.31%, and Claude-Fable-5.1 at 25.22%. Dai said the evaluation images and text had been withheld from training and that checks of the training data found no overlap.
This is not a formal, like-for-like ranking from the paper. Glimpse’s result came from one internal run without an agent or tool harness; the six models in the paper were each run three times and averaged. The public materials do not provide an independent audit or a complete record for replication. So 50.48% is a result worth following up, not proof that Glimpse reliably outperforms the others—and certainly not proof that its visual-thinking mechanism caused the score.
Diagnose the Failure Before Choosing a Fix
For engineering teams, the practical value of a DecoyCount-style evaluation is that it turns “the vision model counts poorly” into more specific questions: Did it miss an object, include a lookalike, misread text, or lose track while enumerating? If different phrasings produce sharply different answers for the same image, teams can investigate whether the priority is annotation and training data, perception, or the counting rule and reasoning strategy—instead of guessing from aggregate accuracy alone.
The benchmark still focuses on visual counting decoys. It cannot stand in for evaluation of broader visual reasoning or deployed tasks. Glimpse is described by Elorian as generating and editing images during reasoning, using a visual workspace and tools, but public information is not sufficient for independent reproduction of that process. A practical decision is to treat its score as a candidate-model signal, repeat the test under matched settings, and inspect how errors shift across decoy types before deciding whether the mechanism merits adoption.