Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence
A study evaluates whether LLM explanations are necessary or sufficient for their decisions using black-box interventions across eight models from Claude, GPT, and Gemini families. The mean Spearman correlation between cited factor rankings and necessity/sufficiency scores is 0.349.
Researchers tested two interpretations of LLM explanations—necessity (changing a factor changes output) and sufficiency (retaining it preserves output)—in synthetic use cases: recommending advisors and judging prompt harmfulness. Models returned outputs and top three influencing factors. Controlled interventions estimated necessity and sufficiency scores. Across eight models, mean Spearman correlations between cited rankings and scores were 0.349 for necessity and an unspecified value for sufficiency.
The low correlation (0.349) suggests LLM explanations often do not align with actual decision behavior, indicating that cited factors may not be causally necessary or sufficient. This challenges the reliability of using explanations for monitoring or error diagnosis in agent workflows.
Organizations relying on LLM explanations for compliance, debugging, or escalation may need to validate explanations behaviorally rather than assume their accuracy. This could drive demand for explanation evaluation tools and standards.
Improved explanation validation can reduce operational risk and increase trust in AI systems, potentially lowering costs associated with incorrect escalations or compliance failures.
Expect further research into causal explanation methods and benchmarks for explanation fidelity. Possible development of intervention-based evaluation frameworks for production LLM systems.