Future AI Pulse
An AI verdict that a research idea is “novel” should be treated as a claim to investigate, not a substitute for a literature comparison. Ai2’s October 27 release introduces a benchmark, data and code for studying novelty judges. Its results show that changing an evaluator’s prompt can substantially change its verdict on the same ideas.
What Ai2 found
In the official study announcement, Ai2 reports that one GPT 5.4 comparison fell from 93.5% to 40.9% accuracy after a prompt change on its Human+Generated setting. This was a particular comparison, not the measured accuracy of every novelty judge.
The benchmark draws novelty signals from ICLR 2026 peer reviews. It distinguishes human authored ideas with stronger or weaker reviewer evidence from a second setting pairing human ideas with outputs from a basic generator. Ai2 explicitly calls the generated ideas’ lower novelty labels weak labels rather than individually verified ground truth.
The team also reports that more search or reasoning did not consistently improve evaluation. Future AI Pulse has not reproduced the results. The release is evidence of sensitivity under the tested conditions, not proof that every AI research assistant is unusable.
Why a plausible explanation is not enough
Imagine a fictional proposal to combine a retrieval system with a new document ranking rule. A judge might praise the combination as original while overlooking a paper that already evaluates the same rule under another name. A confident paragraph would still leave the central evidence question unanswered.
For a researcher, the useful output is a comparison with identifiable prior work: what the earlier method did, what the proposal changes and which claim remains uncertain. A novelty score without that comparison is difficult to challenge or improve.
A practical evidence table for an idea review
- State the contribution narrowly. Write the proposed method, problem and claimed difference in separate sentences. Avoid making “new” the only description.
- Collect the nearest prior work. Record the paper title, date, link and relevant passage. Keep a record of the search terms and sources you checked.
- Compare on a fixed basis. For each source, explain the overlap in method, data, assumptions and evaluation. Distinguish a new application from a new mechanism.
- Preserve unresolved claims. Mark missing papers, ambiguous terminology and claims requiring a domain expert. A failed search is not proof that no related work exists.
- Check sensitivity before trusting a ranking. If a reasonable rephrasing reverses the verdict, record the disagreement and inspect the evidence rather than selecting the more encouraging answer.
This is an editorial review workflow, not a reproduction of Ai2’s experiment. Our guide to finding evidence before summarizing offers a related approach for keeping claims attached to passages a reader can inspect.
An adaptable prompt
Assess this proposed contribution against only the supplied papers. For each paper, identify the closest overlap and cite the passage supporting it. Separate differences in method, application, data and evaluation. Do not infer novelty from missing search results or writing style. List the strongest counterexample to the novelty claim and the additional sources or expert checks needed. Return a provisional comparison, not an acceptance decision.
Supply the proposal and actual source material with the prompt. If the assistant retrieves additional papers, open them and verify that the claimed method or result appears in the source. A citation title alone cannot establish the comparison.
What a useful review can conclude
A defensible result might say that a proposal differs from a specific set of inspected papers in one stated respect, with a remaining uncertainty about terminology or coverage. That conclusion is narrower than “this has never been done,” but it gives the researcher a concrete next step.
Ai2’s release makes evaluator design part of the research question. Teams comparing idea generators should document the judge, prompt, evidence access and evaluation setting alongside their scores. Otherwise, a change in the judging process can be mistaken for progress in the ideas themselves.
Sources and image credit
Ai2: Can we trust LLMs to judge whether a research idea is novel?, published and consulted October 27, 2026. Results and benchmark design are attributed to Ai2. The example, workflow and prompt are Future AI Pulse analysis.
Cover: Yen Vu on Unsplash, under the Unsplash License. Illustrative research workspace, not a screenshot of the benchmark.