通过因果干预分析多流残差结构的信息分配机制。
Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections
- 提出系统性流消融与恢复框架，实现推理时的因果比较。
- 发现并行流间存在非对称信息利用，而非简单冗余。
- 适合研究模型可解释性与深层架构设计的读者。
多流Transformer架构近年来被视为缓解残差连接中表征坍缩和梯度消失问题的有前景方向，但其内部机制尚不明确。特别是新提出的流形约束超连接（mHC）架构采用受控交互的多残差流，却缺乏深入的机理分析。本文首次开源了mHC语言模型（https://huggingface.co/wgpeng/mhc-780m），并通过一系列表示层度量与因果干预手段，探究并行流如何编码与利用信息。具体而言，我们构建了一套系统的流消融与恢复框架，可在推理过程中直接进行因果比较。通过定向成对干预与受控恢复实验，区分了功能冗余与非对称利用，并揭示了信息在流间的分布方式，超越了仅靠表征相似性可观察到的范围。
原文摘要 · Abstract (English)
Multi-stream transformer architectures have recently been proposed as a promising direction for managing representation collapse and the vanishing gradient problem for residual connections, yet their internal mechanisms remain unexplored. In particular, the recently introduced Manifold-Constrained Hyper-Connections (mHC) architecture posits multiple residual streams with constrained interaction, but lacks in-depth mechanistic analysis. We present the first open-source mHC language model (https://huggingface.co/wgpeng/mhc-780m) and analyze the multiple-stream architecture with a suite of representation-level metrics and causal interventions to probe how parallel streams encode and utilize information. Specifically, we introduce a systematic stream ablation-and-rescue framework that enables direct causal comparison of residual streams during inference. Through targeted pairwise interventions and controlled recovery experiments, we distinguish functional redundancy from asymmetric utilization and reveal how information is distributed across streams beyond what is observable from representational similarity alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。