发现多流架构中存在流主导现象，提出初始化对称性破缺可提升性能
Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation
- 通过细粒度诊断发现多流模型实际使用中存在流间信息交换不足
- 多数信号和特征集中于单一主流，多流残差连接利用率低
- 在初始化时打破对称性可缓解流主导问题，提升多种变体性能
超连接（Hyper-Connections, HC）用多条残差流替代单条流，引入流索引的置换对称性。本文研究该对称性在实践中如何被打破：流是否均衡分工，还是出现主导流？通过对基于HC的语言模型进行细粒度诊断，我们追踪了多流表示的实际使用情况。结果发现，在早期播种阶段后，残差混合常保持接近恒等变换，限制了流间信息交换的核心机制。此外，信号与可解释特征高度集中于单一主导流，名义上的多流残差连接未能发挥其容量，表现更接近单流路径。最后，我们证明在流初始化时打破对称性能减少主导行为，并在不同mHC变体中提升性能。代码已公开。
原文摘要 · Abstract (English)
Hyper-Connections (HC) replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices. We study how this symmetry is resolved in practice: whether streams specialize in a balanced way or exhibit dominant-stream usage. Using fine-grained diagnostics for HC-based language models, we trace how multi-stream representations are actually used. We find that after an early seeding stage, residual mixing often remains close to identity, limiting a core HC mechanism for exchanging information between streams. Moreover, both signal and interpretable features concentrate in a dominant stream, and the nominally multi-stream residual connection can underutilize its capacity, behaving closer to a single-stream residual pathway. Finally, we show that breaking symmetry at stream initialization reduces dominant behavior and improves performance across \textit{m}HC variants. Our code is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。