Evidence at a glance
The Answer May Depend on How the Question Is Asked
Ai2 released a study, *Old Ideas, Novel Problems*, alongside a benchmark, data, code, and a pipeline for building evaluation sets. It addresses an increasingly practical question: as systems generate research proposals, can another LLM reliably judge whether those proposals are novel relative to prior work?
The study matters to technical leads not because it offers another model leaderboard, but because it tests the judgment process on which such rankings depend. On the same set of human-and-generated idea pairs, GPT-5.4's accuracy fell from 93.5% to 40.9%—below chance—after a change in prompt wording. The model also changed its choice on more than half of the pairs. If phrasing can push a judge toward the opposite conclusion, a single score is weak evidence that a research system has improved.
The Labels Come from Reviews, Not Acceptance Decisions
The benchmark begins by locating explicit comments on originality in ICLR 2026 reviews, rather than treating acceptance as a proxy for novelty. Its high-novelty pool contains 154 ideas from papers in the top 10% of their research area by score that were accepted and whose reviewers praised novelty without any reviewer disputing it. The low-novelty pool contains 145 ideas from papers in the bottom 10% that were rejected, with reviewers raising originality concerns and none praising novelty.
This is more relevant than using acceptance or rejection alone: a paper may be accepted for other strengths, while a rejected paper may still contain an original contribution. In a check of the review-mining process, an annotator agreed with the direction of the novelty signal in 47 of 50 cases. None of the three disagreements changed whether a review counted as praising or questioning novelty. Still, these labels are evidence grounded in reviewer judgments, not an absolute ruling on what novelty means.
Two Test Sets Ask Different Questions
Human-Only tests whether judges can distinguish the two ends of the human-authored pool. Its evidence is relatively clear, but it also focuses on cases where reviewers expressed a clear view. That is not the same as handling real research proposals whose novelty is mixed or ambiguous. The benchmark can test whether a judge recognizes strong signals, but that alone does not show it can resolve the difficult middle ground researchers often face.
Human+Generated pairs the 154 high-novelty human ideas with 154 ideas produced by a simple LLM. Treating the generated ideas as lower-novelty is a weak-label assumption, not an expert verification of every item. The team blind-reviewed 30 pairs, and experts chose the human idea in every case, but that does not validate the entire generated set. Together, the two settings probe different situations, but the generated ideas' label strength should not be conflated with papers whose reviewers explicitly discussed novelty.
Prompts Are Part of the Evaluation Protocol
The main experiments compared GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. The researchers varied prompts, access to relevant papers, reasoning budgets, idea formats, and the number of votes. The result is not simply that more context improves accuracy: retrieval and additional reasoning did not help consistently, and models differed in how they responded to prompt changes. The evaluation setup is therefore not packaging around the model; it is a variable that shapes the score.
That has direct consequences for development workflows. If a team compares two research systems using a single prompt, the measured difference may partly reflect the judge's protocol rather than the quality of the candidate ideas. When AI-generated work is scored by another model, teams should record how the task is framed, what comparison rules apply, and what evidence the judge receives. They should also check whether the conclusion survives a reasonable rephrasing.
A More Elaborate Judge Is Not Automatically More Trustworthy
In Ai2's specific Human+Generated pointwise comparison, Claude Opus 4.6 with three votes achieved a macro-F1 of 0.854 at a cost of $2.59. Idea Novelty Checker scored 0.642 at $30.66, while Sakana AI Scientist scored 0.772 at $82.05. In a separate extension using GPT-5.6 Sol with a high reasoning budget and iterative retrieval, macro-F1 was 0.78—11 points below Claude Opus 4.6—with costs about ten times higher. These figures describe particular test settings, not a universal ranking across tasks. They do, however, show that adding retrieval, reasoning, and specialized workflows does not guarantee an improvement over a simple baseline.
Automated novelty judges are therefore better treated as controlled evaluation components in research systems than as authorities that decide which directions to pursue. Before deployment, teams should fix and disclose the evaluation protocol, report sensitivity to prompt changes, and state where generated-idea labels come from and how uncertain they are. The benchmark focuses on ICLR 2026 and cases at the clearer ends of the novelty spectrum. It does not prove that every model fails across every discipline. Its more actionable conclusion is narrower: establish that a judge is stable before using it as evidence that a system has produced something genuinely new.