Researchers gave AI models the exact sources scientists used for a discovery, and asked it to guess the finding. It got it right three percent of the time.
Tech Times covered a new contamination-resistant benchmark called "Reconstruction," and the results are humbling for AI's science credibility.
How the Test Was Actually Built
The benchmark's design is deliberately strict, and that's the whole point. Researchers pulled 643 real research papers across six scientific domains, and for each one, gave an AI model only the bibliography -- the reference list the paper's authors had cited before their own paper was published -- with the paper itself, any later citing literature, and any post-publication signals stripped out entirely. That "blind" setup exists specifically to rule out the AI simply having memorized the paper's actual finding from its training data, a known contamination problem that's made a lot of prior AI-science benchmarks hard to trust. An independent AI judge then compared each model's proposed hypothesis against the real, held-out finding to score a match.
The Ceiling
Every frontier large language model tested -- seven of them -- failed to reconstruct a research paper's core finding from its reference list alone more than fifteen percent of the time, and most scored far lower, hitting a floor of just three percent.
The Team Effort
Researchers also tried a multi-agent tournament: having AI models compete and critique each other's hypotheses in a structured, Swiss-tournament-style format, without any external web search to lean on. The outcome: twenty-three to forty-two percent match rates, roughly a 2.4-times lift over any single model working alone -- a real improvement, but still leaving most ideas unrecovered.
The Telling Part
No single model distinguished itself dramatically at the top. That near-uniformity of failure across otherwise capable systems rules out explanations centered on prompting strategy or model size -- if a smarter model or a cleverer prompt could fix this, someone would likely have found it by now. The gap appears to be structural: today's models are consistently much better at summarizing and connecting existing information than at the specific inferential leap of proposing something genuinely new from partial evidence.
The Industry Implication
A significant industry of "AI scientist" products now markets large language models as hypothesis-generation engines. Many of those claims rest on evaluations that let the model see the full paper text, author information, or post-publication signals -- exactly the conditions this benchmark deliberately excludes. That gap between how these products are typically evaluated and how this benchmark tests them is itself worth noting: a much easier test tends to produce a much rosier result.
The Takeaway
A model that's a great literature summarizer may perform far worse when actually asked to do what scientists prize most: forming a genuinely novel hypothesis from available evidence, rather than recognizing or recombining something it's already seen.