BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
BiG-SURE is an uncertainty estimator based on cross-temperature semantic agreement. It samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. The method is evaluated on text QA, multilingual QA, and multimodal QA tasks across multiple model families, improving average abstention AUROC over prior black-box uncertainty estimators while remaining simple, unsupervised, and applicable to black-box models.
BiG-SURE uses cross-temperature sampling to create semantic anchors and probes, then builds a bipartite graph with NLI entailment scores. Confidence is derived from the normalized squared spectral energy of the anchor-probe matrix, providing a spectral measure of semantic agreement. This approach is unsupervised and works in black-box settings, making it applicable to LLMs and VLMs without parameter access.
Reliable uncertainty estimation is critical for deploying LLMs and VLMs in safety-critical applications. BiG-SURE offers a black-box, unsupervised method that improves abstention AUROC over prior estimators, potentially lowering barriers for enterprises to adopt LLMs in high-stakes domains where model internals are not accessible.
BiG-SURE provides a practical uncertainty estimation tool for black-box LLM deployments, enabling safer and more reliable use in regulated industries such as healthcare, finance, and autonomous systems. Its unsupervised nature reduces integration cost and avoids the need for model retraining or access to internal parameters.
Future work may extend BiG-SURE to other modalities and model families, validate it on additional safety-critical tasks, and explore integration with deployment pipelines for real-time uncertainty monitoring. Adoption in commercial LLM APIs could follow if the method demonstrates consistent gains across diverse black-box models.