Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity
Instruction-tuned language models exhibit verbalized overconfidence in question answering. Instruction tuning consistently alters answer confidence despite limited changes in predictive accuracy and decreases in likelihood-based calibration. Cross-rationale diversity consistently decreases after instruction tuning, while surface-level lexical diversity varies in direction and magnitude across models and benchmarks. These differences persist after controlling for answer selection and rationale length.
A study evaluates three matched base and instruction-tuned models across question-answering benchmarks. It finds that instruction tuning consistently alters answer confidence, with limited changes in predictive accuracy and decreases in likelihood-based calibration. The effect on rationale diversity is non-uniform: cross-rationale diversity consistently decreases, whereas surface-level lexical diversity varies. These differences persist after controlling for answer selection and rationale length, indicating that confidence and rationale diversity capture distinct effects of instruction tuning.
Instruction tuning decouples verbalized confidence from predictive accuracy and likelihood-based calibration. The consistent decrease in cross-rationale diversity suggests instruction tuning narrows the semantic space of generated rationales, while surface-level lexical diversity changes are model- and benchmark-dependent. Controlling for answer selection and rationale length confirms that confidence and rationale diversity are distinct axes affected by instruction tuning.
The findings imply that instruction-tuned models may express confidence that is not aligned with actual correctness, which could mislead users in high-stakes applications. The reduction in cross-rationale diversity may limit the ability to detect uncertainty through rationale variation. Developers should consider additional calibration or uncertainty estimation methods when deploying instruction-tuned models.
For enterprises using instruction-tuned models in question answering, this research highlights a risk of overconfident outputs that may require mitigation. Products that rely on model confidence for decision support or automated actions could face reliability issues. There may be market demand for calibration services or fine-tuning approaches that restore confidence alignment.
Future work may explore methods to recalibrate instruction-tuned models or design training objectives that preserve rationale diversity. Observing whether these effects scale with model size or persist across different instruction-tuning datasets will be important. If verbalized overconfidence remains uncorrected, it could drive adoption of external calibration tools or post-hoc uncertainty quantification in production systems.