TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents
TruthInsightBench is a benchmark configured for discovery, consisting of 40 blind tasks drawn from 40 peer-reviewed studies across 10 scientific domains. Tasks expose only a neutral scientific objective and frozen data, withholding source conclusions, expected values, and analysis paths. A fixed LLM-based judge scores the evidentiary maturity of an agent's claims along six dimensions, operationalized as 29 artifact-grounded items, with automated deterministic aggregation and no per-instance human grading. On one frozen base model, four coding agents scored between 58.4 and 60.3 out of 100, forming a narrow plateau with no statistically reliable pairwise separation.
TruthInsightBench is a benchmark designed to evaluate autonomous coding agents as AI-scientist systems in open-ended scientific discovery. Unlike existing benchmarks configured for reproduction, TruthInsightBench withholds source conclusions and expected values, requiring agents to determine what claims the data support. The benchmark includes 40 blind tasks from peer-reviewed studies across 10 domains. A fixed LLM-based judge scores claims on six dimensions via 29 artifact-grounded items, enabling automated, repeatable evaluation. Initial results show four coding agents on one frozen base model achieve scores between 58.4 and 60.3, with no statistically reliable pairwise separation, indicating they execute and document analyses competently but have comparatively limited discovery capability.
The benchmark's automated LLM-based judge uses 29 artifact-grounded items across six dimensions to assess evidentiary maturity, enabling deterministic aggregation without human grading. The narrow score plateau (58.4-60.3) suggests current coding agents share similar limitations in open-ended discovery, likely due to reliance on prescribed analysis patterns rather than novel hypothesis generation. Future signals include whether agents improve on TruthInsightBench with different base models or agent architectures, and whether the benchmark's blind task design influences agent development toward more autonomous scientific reasoning.
TruthInsightBench addresses a gap in evaluating AI-scientist systems by focusing on discovery rather than reproduction. This could shift industry benchmarks toward measuring genuine scientific contribution, impacting how AI research tools are developed and marketed. The lack of statistically reliable separation among four coding agents indicates that current commercial and open-source agents may not yet differentiate in true discovery tasks, potentially affecting investment and adoption decisions in AI-for-science.
TruthInsightBench provides a standardized, automated way to evaluate AI agents for scientific discovery, which could help enterprises and research institutions select or develop tools for data-driven research. The benchmark's focus on evidentiary maturity may drive demand for agents that can produce defensible, novel claims, potentially creating new market opportunities in AI-assisted research and development.
If agents begin to show statistically significant improvements on TruthInsightBench, it may signal progress toward autonomous scientific discovery. Conversely, persistent plateauing could highlight fundamental limitations in current LLM-based coding agents. The benchmark's automated, repeatable evaluation may become a standard for tracking progress in AI-scientist systems, influencing research directions and funding.