Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning
A research paper proposes knowledge-aligned supervised fine-tuning (SFT) to reduce hallucinations by constraining training targets to the base model's parametric knowledge. It compares existing generation-based and estimation-based methods and introduces Evidence Rewrite and Recall Rewrite. Experiments with Qwen 3 4B and OLMo 3 7B show reduced factual hallucinations on WildHalu and Biography while largely preserving general capabilities. Recall Rewrite yields the strongest factuality gains and improves refusal behavior on UnknownBench.
The paper studies hallucinations caused by SFT targets that require knowledge not robustly internalized by the base model. It frames mitigation methods as knowledge-aligned SFT and introduces two new variants: Evidence Rewrite, which verifies base-model generations using external evidence, and Recall Rewrite, which retains claims only when consistently recalled by the base model. Experiments demonstrate that knowledge-aligned SFT reduces factual hallucinations while preserving general capabilities, with Recall Rewrite providing the strongest factuality gains and improved refusal behavior.
Knowledge-aligned SFT constrains training targets to the base model's parametric knowledge, reducing hallucinations. Recall Rewrite retains only claims consistently recalled by the base model, leading to strongest factuality gains and better refusal on UnknownBench. Evidence Rewrite uses external evidence to verify base-model generations. Both methods are evaluated on Qwen 3 4B and OLMo 3 7B, showing reduced hallucinations on WildHalu and Biography while preserving general capabilities.
This research addresses a key challenge in deploying LLMs: hallucination reduction without sacrificing general capabilities. Knowledge-aligned SFT offers a practical approach for improving factual reliability in fine-tuned models, which is critical for enterprise and consumer applications requiring trustworthy outputs. The methods could be integrated into existing fine-tuning pipelines to enhance model safety and reliability.
Reducing hallucinations while preserving general capabilities can increase trust in AI systems, enabling broader deployment in high-stakes domains such as healthcare, finance, and legal services. Knowledge-aligned SFT may lower the cost of post-training factuality improvements and reduce the need for extensive human oversight, offering a competitive advantage to model providers and enterprises.
Future work may explore scaling knowledge-aligned SFT to larger models and diverse domains, as well as combining it with other hallucination mitigation techniques. Adoption of such methods could become standard practice in fine-tuning workflows, especially for applications where factual accuracy is paramount. Further research may refine the estimation of parametric knowledge boundaries and improve the efficiency of recall-based filtering.