Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO
A study compared nine post-training arms (GRPO, SFT, DPO) from one starting checkpoint, with key comparisons extended across scales and families. Across five tested GRPO variants, grounding gains are small. Conflict-SFT improves grounding moderately, while DPO drives grounding near ceiling on its matched distribution. Conflict-SFT and DPO largely use the same causal attention-head set as the starting model. Subtracting the starting-model direction suppresses both gains, while adding it to the starting model recovers 35% of DPO's gain.
Language models can ignore prompt evidence when it conflicts with memorized knowledge. Post-training can make models follow such evidence more reliably, but it is unclear whether these gains require new machinery or strengthen machinery already present. The study compares nine post-training arms spanning GRPO, SFT, and DPO from one starting checkpoint, with key comparisons extended across scales and families. It estimates a grounding direction from that checkpoint before training. Across five tested GRPO variants, grounding gains are small. For the two variants replicated across seeds, equivalence tests bound their effects below the conflict-SFT gain even as the rewarded metric improves. Conflict-SFT improves grounding moderately, while DPO drives grounding near ceiling on its matched distribution. Conflict-SFT and DPO largely use the same causal attention-head set as the starting model. Subtracting the starting-model direction suppresses both gains, while adding it to the starting model recovers 35% of DPO's gain at a dose passing all stated side-effect checks. After a supervised warm start makes the context answer appear in more rollouts, the same GRPO recipe adds essentially no further grounding.
The study suggests that context-grounding gains from post-training are largely mediated by pre-existing model machinery rather than newly created circuits. The finding that subtracting the starting-model direction suppresses gains and adding it recovers a significant fraction of DPO's gain indicates that post-training primarily amplifies or reweights existing attention-head pathways. The small gains from GRPO variants, despite improvements in the rewarded metric, highlight a potential disconnect between reward optimization and actual grounding behavior.
This research has implications for how AI labs approach post-training for instruction following and factuality. If grounding gains are mediated by pre-existing machinery, then targeted interventions on specific attention heads or directions may be more efficient than broad post-training. The limited effectiveness of GRPO for grounding, despite its popularity in reinforcement learning, suggests that labs should carefully evaluate whether RL-based methods actually improve desired behaviors beyond surface metrics.
For AI product developers, this research suggests that improving a model's ability to follow user-provided context may be achievable with less compute-intensive methods than full RL pipelines. The finding that DPO can drive grounding near ceiling on matched distributions indicates that preference-based methods may be a cost-effective route to better instruction adherence. However, the limited gains from GRPO imply that investments in RL infrastructure for grounding may not yield proportional improvements.
Future work may focus on identifying and directly manipulating the grounding direction in models to achieve more reliable context adherence. The study's approach of estimating a grounding direction before training could become a diagnostic tool for evaluating post-training methods. Further research may explore whether similar pre-existing machinery mediates other capabilities, such as reasoning or safety behaviors.