How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
A study examines how trained models use the widened residual pathway of Hyper-Connections and its manifold-constrained variant mHC, focusing on DeepSeek-V4-Flash's four-stream residual pathway. It finds that read/write routing is concentrated but varies across depth, with a typical attention or FFN site effectively using about two streams. Residual mixing is modest and occurs primarily in early layers; in layers 22-42, the pathway mostly carries each stream forward separately. Replacing late mixers with identity increases C4 perplexity by only 1.9% and preserves six-task average score, while replacing early mixers increases perplexity by 41%.
The paper investigates how DeepSeek-V4-Flash utilizes its four-stream residual pathway from Hyper-Connections/mHC. It reports that routing is selective, with about two streams effectively used per site, and that residual mixing is concentrated in early layers. Late-layer mixers are near-identity, as replacing them with identity has minimal impact on perplexity and task performance, whereas early mixers are functionally important.
The evidence suggests that the residual pathway in mHC is not uniformly utilized; instead, it exhibits depth-dependent specialization. Early layers perform significant cross-stream mixing, while later layers maintain near-identity mixing, implying that the model learns to route information selectively and that the additional streams may serve distinct representational roles. The low impact of replacing late mixers with identity indicates redundancy or minimal contribution of mixing in those layers, possibly due to already separated representations.
This research provides insight into the internal mechanics of large language models using Hyper-Connections, which could inform more efficient architecture designs. Understanding that late-layer mixing is near-identity may allow for pruning or simplifying those components without significant performance loss, potentially reducing computational overhead in deployed models.
If late-layer mixers can be replaced with identity operations with minimal performance degradation, this could lead to reduced inference costs and memory usage for models employing Hyper-Connections, offering potential efficiency gains for AI service providers.
Future work may explore whether these findings generalize to other models and tasks, and whether architectural modifications based on selective routing can improve efficiency. The observed functional significance of early mixers suggests that targeted interventions could be used to manipulate model behavior or interpretability.