StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions
Large language models often answer the same multiple-choice question inconsistently when posed under support-oriented and elimination-oriented framings. A dual-framing protocol with minimally varied prompts was introduced. An untrained special token, [STATE], was appended and its residual-stream activation used as an intervention interface. Across both models, the two framings induce separable [STATE] activations concentrated in intermediate layers. Swapping these activations between paired prompts systematically changes predictions and improves cross-framing agreement. Mean-difference steering directions derived from the dual-framing contrast exhibit more bounded layer-wise responses than matched contrastive activation addition directions under the evaluated protocol.
Researchers introduced StateSwap, a method to probe hidden states in large language models when answering multiple-choice questions under support-oriented versus elimination-oriented framings. By appending a special [STATE] token and swapping its activations between paired prompts, they showed that the two framings induce separable internal representations in intermediate layers. Swapping these activations systematically changes predictions and improves cross-framing agreement, providing intervention-based evidence that the activations are behaviorally relevant. The study also found that mean-difference steering directions from the dual-framing contrast exhibit more bounded layer-wise responses than contrastive activation addition directions.
The use of an untrained [STATE] token as an intervention interface allows direct manipulation of residual-stream activations without modifying model weights. The finding that support- and elimination-oriented framings produce separable activations in intermediate layers suggests that framing-specific computations are localized and can be transferred between prompts. The bounded layer-wise responses of mean-difference steering directions indicate a more controlled intervention compared to contrastive activation addition, potentially reducing unintended side effects.
This research highlights a potential source of inconsistency in LLM outputs when the same question is framed differently. For applications requiring reliable multiple-choice reasoning (e.g., standardized testing, medical diagnosis support, legal analysis), understanding and mitigating framing-induced variability could improve answer consistency. The intervention technique may inspire new methods for controlling model behavior without fine-tuning.
Improving cross-framing consistency in LLMs could enhance trust in AI systems used for high-stakes decision-making. The intervention method offers a lightweight, post-hoc way to align model behavior without retraining, potentially reducing deployment costs. Companies developing LLM-based products for education, legal tech, or healthcare may benefit from more reliable multiple-choice answering.
Future work may explore whether similar hidden-state interventions can be applied to other task framings or modalities. The approach could be extended to larger models and more diverse question types. If the technique proves robust, it might be integrated into model serving pipelines to enforce consistent reasoning across prompt variations. Further research is needed to assess generalization beyond the evaluated models and protocols.