arXiv:2609.05309cs.LGcs.AI2026-09

探究大模型中多流残差路径如何被使用,发现早期混合有限、后期基本保持独立。

How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing

论文配图:How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
图 1 · 摘自论文原文
  • 通过有效流数与跨流权重分析,发现每层仅约两流被实际使用。
  • 早期层有适度残差混合,晚期层主要保持各流独立传递。
  • 关键模块结构重要性高于参数波动,适合研究模型内部路由机制者阅读。

超连接及其流形约束变体mHC在一条残差路径中连接四条流,但其训练后如何利用该能力尚不明确:各模块读写范围如何、残差路径混合强度如何、各流是否携带不同表征。本文以DeepSeek-V4-Flash的四流残差路径为研究对象,采用有效流数、跨流残差权重和流间余弦相似度进行分析。结果表明,读写路由集中但随深度变化:典型注意力或前馈层有效使用约两流,主导流在层间变化,且各流表征方向保持区分。残差混合程度温和,主要集中于早期层;第22至42层中,路径基本独立传递各流。干预实验显示功能意义:用恒等映射替换晚期混合器仅使C4困惑度上升1.9%,六任务平均得分保持不变;而替换早期混合器则导致困惑度上升41%。将每个早期混合器固定为C4诊断均值,困惑度仅上升0.2%,平均得分下降0.25个百分点,说明站点特定结构比令牌级变化更重要。保留每令牌最大三权重,困惑度最多上升2.7%,得分变化不超过0.4分。因此,该模型并未充分实现四流mHC的全部灵活性:单个模块极少需四流,且晚期残差混合无显著测量收益。

原文摘要 · Abstract (English)

Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n, yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations. We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. Read/write routing is concentrated but varies across depth: a typical attention or FFN site effectively uses about two streams, while the dominant stream changes across layers and the representations remain directionally distinct. Residual mixing is modest and occurs primarily in early layers; in layers 22-42, the pathway mostly carries each stream forward separately. Targeted interventions establish the functional significance of these patterns. Replacing the late mixers by identity increases C4 perplexity by only 1.9% and preserves the six-task average score, whereas replacing the early mixers increases perplexity by 41%. Fixing each early mixer to its C4 diagnostic mean increases perplexity by only 0.2% and reduces the average score by 0.25 percentage points, showing that its site-specific structure matters more than its token-wise variation on the evaluated metrics. Likewise, retaining the three largest routing weights per token at every site increases perplexity by at most 2.7% and changes the average score by at most 0.4 points. Thus, the studied model realizes only part of the flexibility afforded by four-stream mHC: individual blocks rarely require all four streams, and late residual mixing provides little measured benefit.

残差路径多流架构模型路由深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。