A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations: I. The Single-Block Theory and Synthetic Validation
A paper on arXiv (2608.03620v1) studies when activation patching and weight-space ablation agree on causal responsibility for a behavior. It analyzes an idealized model with additive conditional computation in a residual stream and proves three exact results: (1) deleting a subset of carriers collapses a matched input pair onto the same unconditional output iff the removal is symmetric and leaves no outside contrast; (2) patching a carrier moves the readout by donor-receiver contrast, while ablation moves it by absolute level, and neither bounds the other; (3) for an attention head composed with its own layer normalization and MLP, an exact first-order interaction formula with provably second-order remainder is derived.
A new theoretical paper examines the relationship between activation patching and weight-space ablation, two methods for attributing causal responsibility in neural networks. The authors prove that under an idealized additive residual stream model, the two methods can disagree: patching depends on contrast between donor and receiver, while ablation depends on absolute level. They provide exact conditions for when deleting carriers causes output collapse and derive an interaction formula for attention heads with layer normalization and MLPs.
The paper formalizes the discrepancy between activation patching and weight-space ablation by showing that patching measures contrast while ablation measures absolute level, implying that single-carrier patching can flip decisions even when no single-carrier ablation does. The derived first-order interaction formula for attention heads with layer norm and MLP provides a precise decomposition with a second-order error bound, which could guide more accurate causal attribution in transformer circuits.
This work addresses a fundamental methodological question in mechanistic interpretability, which is increasingly important for debugging and aligning large language models. By clarifying when patching and ablation agree, it may lead to more reliable tools for identifying and modifying specific behaviors in production models, potentially reducing unintended side effects during model editing or safety interventions.
Improved causal attribution methods can enhance model transparency and trustworthiness, which is critical for enterprise adoption in regulated industries. More reliable model editing could reduce the cost of fixing undesirable behaviors without full retraining, offering a competitive advantage for AI platform providers.
Future work may extend the theory to multi-block architectures and validate it on real transformer models. If the interaction formula holds empirically, it could enable more precise circuit discovery and targeted model editing. Observers should watch for follow-up papers applying these results to interpretability tooling or safety evaluations.