Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generation
A study on arXiv evaluates whether hints that turn failing generated programs into passing ones provide missing information or merely steer the model toward a solution it could already produce. Tests on HumanEval+ and MBPP+ with Qwen2.5-3B-Instruct and Phi-3.5-mini show that relevant hints rescue some failures, but many of those rescues are also achieved by unhinted sampling, suggesting hints may not teach new skills. Mechanistic tests find a stable activation direction shared by relevant and unrelated hints, with no detectable net accuracy gain from adding this direction.
The paper 'Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generation' investigates whether hints that fix failing code generations actually impart new capabilities or just guide the model to solutions it could already produce. Using executable evaluation on HumanEval+ and MBPP+, the authors test Qwen2.5-3B-Instruct and Phi-3.5-mini. For Qwen2.5-3B-Instruct, adaptive relevant hints rescue 36 of 79 selected failures, while an unrelated hint rescues 19; unhinted sampling solves 46 and recovers 31 of the 36 relevant-hint rescues. Phi-3.5-mini shows a similar pattern: relevant hints rescue 42 of 101 failures, unrelated hints rescue 17, and unhinted sampling solves 57, including 36 of the 42 relevant-hint rescues. Because hint conditions use different attempt budgets, the comparisons do not isolate a purely semantic effect. Mechanistic tests on Qwen identify a stable activation direction shared by relevant and unrelated hints; persistently adding this direction yields 14 rescues and 18 regressions with no detectable net accuracy gain. Learned low-rank interventions have a positive but imprecise estimated effect. Full textual specifications solve 22 of 24 context-defined problems, versus 5-11 for tested virtual-KV prefixes.
The study uses mechanistic interpretability to identify a shared activation direction for relevant and unrelated hints, suggesting that hints may act as a generic steering signal rather than providing task-specific information. The lack of net accuracy gain from adding this direction implies that the model's failures are not due to missing knowledge but to sampling or decoding issues. The comparison between full textual specifications and virtual-KV prefixes indicates that explicit context is more effective than internal state manipulation for code generation.
This research suggests that prompt engineering with hints may not reliably improve code generation models' underlying capabilities; instead, it may only bias sampling toward already-known solutions. For AI-assisted coding tools, this implies that simply providing hints or partial specifications may not teach models new skills, and developers may need to rely on more explicit context or different training approaches to improve performance on novel problems.
For companies building AI coding assistants, this research highlights the limitations of hint-based prompting and suggests that investment in better context provision (e.g., full specifications) may yield more reliable improvements than hint engineering. It also indicates that evaluating models on unhinted sampling performance may be a more accurate measure of true capability, which could influence model benchmarking and selection.
Future work may focus on distinguishing between semantic and steering effects of hints by controlling attempt budgets, and on developing interventions that genuinely transfer skills rather than just bias sampling. The finding that full textual specifications outperform virtual-KV prefixes could lead to more effective context-engineering methods for code generation. Additionally, the use of mechanistic probes may help identify when a model lacks knowledge versus when it fails to apply existing knowledge.