Attention-Path Fragility as an Uncertainty Signal in Large Language Models
A training-free estimator, ASMI (Attention-Subnetwork Mutual Information), masks attention heads and measures BALD mutual information among subnetworks with a semantic-agreement kernel. On grounded QA, out-of-fold testing shows it adds error-predictive information beyond single-pass confidence and entropy, concentrated in confident-but-fragile predictions where acting on it roughly halves the retained error of a confidence filter. Sem-ASMI reads the signal from a single greedy response and ties or beats Semantic Entropy on ten of twelve benchmarks.
Researchers propose that model uncertainty can be detected by the fragility of confident predictions under perturbation of attention pathways. The method, ASMI, uses a semantic-agreement kernel to measure mutual information among masked attention subnetworks, providing an uncertainty signal that is distinct from output confidence. It is particularly effective for grounded QA, where answers depend on provided context, and can be obtained from a single greedy response, matching or exceeding Semantic Entropy on most tested benchmarks.
ASMI leverages attention-head masking to create subnetworks and computes BALD mutual information with a semantic-agreement kernel, capturing uncertainty not reflected in output probabilities. The signal is regime-graded, strong when answers are routed through context and bounded when recalled from parametric knowledge, indicating it predicts its own domain of applicability.
This approach could improve reliability of LLM deployments in high-stakes applications by identifying confident-but-fragile predictions, enabling selective human review or fallback mechanisms without requiring multiple stochastic generations.
Enhances trustworthiness of LLM outputs in enterprise settings by reducing errors in confident predictions, potentially lowering risk in customer-facing or decision-support systems.
Further validation on diverse tasks and integration into production systems may follow. The method's training-free nature and single-pass requirement make it attractive for real-time applications, but its dependence on attention architecture may limit applicability to transformer-based models.