通过约束残差流混合提升状态空间语言模型性能，兼顾稳定与效率。
mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models with Stream-Specialized Adapters
- 用双随机矩阵流形约束多流残差混合，增强训练稳定性。
- 在WikiText-2上验证损失降至6.1353，困惑度降为461.88，显著提升生成质量。
- 轻量适配器设计适合需要高效微调的场景，适合部署优化研究者。
mHC（流形约束超连接）通过Sinkhorn-Knopp投影将多流残差混合矩阵约束于双随机矩阵流形，提升稳定性。本文将该机制引入状态空间模型（SSM）语言建模，在SSM块中扩展残差流为多并行流，通过单纯形约束预混合聚合为单一SSM输入，再经单纯形约束后混合将输出散射回各流，并在每层应用Sinkhorn投影的残差混合。此外引入流专用适配器，通过共享瓶颈与流级缩放实现轻量级容量扩充，分别部署于流聚合前及SSM输出后。在相同训练设置下对WikiText-2进行评估，静态mHC使验证损失从6.3507降至6.2448，困惑度从572.91降至515.35；加入适配器后进一步降至6.1353和461.88。性能提升伴随吞吐量小幅下降（1025.52→938.90 tokens/s）和峰值显存上升（2365→3092 MB），表明该方法可在可控开销下有效提升模型质量。
原文摘要 · Abstract (English)
Manifold-Constrained Hyper-Connections (mHC) introduce a stability-motivated variant of multi stream residual mixing by constraining residual stream mixing matrices to the manifold of doubly stochastic matrices via Sinkhorn-Knopp projection. In his work, we study whether mHC-style constrained multi-stream residual topology transfers effectively to state space model (SSM) language modeling. We implement a static mHC mechanism around an SSM block by expanding the residual stream into multiple parallel streams, aggregating streams into a single SSM input through simplex-constrained pre-mixing, scattering the SSM output back to streams through simplex-constrained post-mixing, and applying Sinkhorn-projected residual stream mixing at each layer. We further introduce stream-specialized adapters that add lightweight stream-specific capacity through a shared bottleneck with per-stream scaling, applied both before stream aggregation and after the SSM output prior to scattering. We evaluate baseline single-stream SSM, static mHC SSM, and mHC SSM with adapters on WikiText-2 using identical training settings and report checkpoint-based validation loss, perplexity, throughput, and peak GPU memory. Under the reported fair checkpoint evaluation, static mHC improves validation loss from 6.3507 to 6.2448 and reduces perplexity from 572.91 to 515.35, while mHC with adapters further improves validation loss to 6.1353 and perplexity to 461.88. These gains are accompanied by modest throughput reductions from 1025.52 to 964.81 and 938.90 tokens per second, and increased peak memory from 2365 MB to 2568 MB and 3092 MB. The results suggest that mHC-inspired constrained multi-stream residual mixing can yield measurable quality improvements in SSM language models and that stream-specialized adapter capacity can further enhance performance with predictable efficiency tradeoffs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。