Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies
A new benchmark called Reconstruction tests whether language models can recover the true research idea of a published paper given only its pre-publication bibliography. The benchmark withholds the seed paper and all contemporaneous or future literature, and uses an independent LLM judge to match proposed hypotheses against the held-out ground-truth idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve Match rates of approximately 3-15%. A reference-only multi-agent pipeline combining cross-model review with a Swiss tournament over aligned hypothesis slots raises Match rates to approximately 23-42%, an observed 2.4x lift over the best single-model baseline.
The Reconstruction benchmark evaluates the ability of language models to recover research ideas from pre-publication bibliographies. It employs a strict anti-leakage protocol including temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies. Seven frontier models achieve modest Match rates of 3-15% across six scientific domains and 643 papers. A multi-agent pipeline with cross-model review and Swiss tournament selection improves Match rates to 23-42%, a 2.4x lift over the best single-model baseline. The work is reported as an arXiv preprint.
The benchmark's anti-leakage design prevents prompt-time leakage of the seed idea by withholding the seed paper and contemporaneous literature, using anonymous reference IDs and frozen bibliographies. The multi-agent pipeline uses cross-model review and a Swiss tournament over aligned hypothesis slots, without external web search, to achieve a 2.4x improvement in Match rates. This suggests that ensemble and tournament-based selection mechanisms can significantly enhance idea recovery in language models.
The results indicate that current frontier models have limited ability to infer research ideas from bibliographies alone, with single-model Match rates below 15%. The multi-agent approach demonstrates a practical method to improve performance without external data, which could be relevant for research assistance tools, literature-based discovery, and competitive intelligence applications.
The benchmark and pipeline could inform the development of AI tools for research ideation, patent analysis, and technology forecasting. The demonstrated improvement from multi-agent collaboration may guide investment in ensemble methods for knowledge-intensive tasks.
Future work may explore scaling the multi-agent pipeline, incorporating additional domains, or refining the judge model. Observable next signals include follow-up papers applying the Reconstruction benchmark to new models, or adaptations of the tournament selection method to other reasoning tasks.