用可学习的残差路由替代固定连接，让冻结模型更高效微调。
Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
- 提出可学习的残差路由模块，替代传统Transformer中的固定连接
- 在1B和7B模型上，组合使用mHC与LoRA能降低语言建模损失
- 适合追求低参数量微调且关注残差结构创新的研究者
大多数参数高效微调（PEFT）方法仅调整权重或激活值，而保持Transformer中的残差连接不变。本文提出流形约束的超连接（mHC），作为残差连接的泛化形式，将冻结的OLMo-2主干网络与可学习的残差路由模块结合。研究发现，mHC可在不更新主干的情况下实现微调，但其作用机制与预训练阶段有本质差异：在微调中，将残差混合矩阵固定为恒等矩阵反而能提升性能。作为独立方法，mHC并未持续优于LoRA；但在相同可训练参数预算下，mHC+LoRA组合在1B和7B规模模型上均表现出更低的语言建模损失，并在多个基准任务上取得任务依赖性提升。结果表明，残差路由是一个独特且有前景的新型参数高效微调方向。
原文摘要 · Abstract (English)
Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections. This paper investigates Manifold-Constrained Hyper-Connections (mHC), a generalisation of residual connections, as a novel PEFT approach, wrapping frozen OLMo-2 backbones with learned residual routing modules. We find that mHC can finetune frozen Transformers, but that its role differs fundamentally from the original pre-training setting: in finetuning, fixing the residual mixing matrix to identity often improves performance. As a standalone PEFT method, mHC does not consistently outperform LoRA. However, at matched trainable parameter budgets, mHC+LoRA combinations improve language-modelling loss and show task-dependent benchmark gains at both 1B and 7B scale. Overall, our results identify residual routing as a distinct and promising novel PEFT axis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。