DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction
DiagChain is a diagnostic benchmark for evidence-grounded attack chain reconstruction that enables stage-wise evaluation of LLM agents. It includes MAIN-69, a suite of 69 scenarios spanning multiple operating systems, evidence noise levels, and chain lengths. It introduces Evidence-Centric Retrieval-Augmented Generation (ECRAG), which couples evidence retrieval with an evolving structured representation of the reconstructed chain. Five complementary metrics assess distinct stages of the reconstruction process. Evaluations using 6 LLMs show that even the strongest configuration succeeds on only 39.6% of the 849 reference steps in MAIN-69. Smaller models struggle with more basic tasks.
DiagChain introduces a stage-wise evaluation framework for LLM agents performing attack chain reconstruction, moving beyond aggregate accuracy to diagnose errors at intermediate reasoning stages. The ECRAG method integrates evidence retrieval with a dynamic structured representation, and the five complementary metrics provide granular insight into failure modes. The low success rate (39.6%) of the best configuration highlights significant challenges in evidence grounding and multi-step reasoning for current LLMs.
The benchmark reveals that current LLM agents are not yet reliable for autonomous attack chain reconstruction in cybersecurity operations. The gap between model capabilities and the demands of evidence-grounded reasoning suggests that human-in-the-loop systems will remain necessary. The diagnostic approach may influence how security vendors evaluate and improve AI-driven threat analysis tools.
For cybersecurity companies, DiagChain provides a rigorous evaluation framework to benchmark and improve AI-based attack reconstruction products. It highlights a clear performance ceiling, indicating market opportunities for specialized models or hybrid systems. Enterprises adopting such tools can use the benchmark to assess vendor claims and understand the limitations of current AI in security operations.
Future work may focus on improving evidence retrieval and reasoning fidelity in LLM agents for cybersecurity. The benchmark could be expanded with more diverse attack scenarios and adversarial noise. Progress on DiagChain metrics may become a standard for measuring agentic reasoning in security contexts, and improvements could lead to more autonomous incident response systems.