arXiv:2608.07851cs.LGcs.CL2026-08

提出轻量级路由机制,让深度网络更高效表达动态信息流。

TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing

论文配图:TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing
图 1 · 摘自论文原文
  • 用张量分解压缩路由参数,避免传统方法的参数爆炸
  • 8路残差时性能超越现有方法,参数仅需其16%
  • 结构清晰可解释,适合追求效率的模型设计者

残差连接依赖静态路径,而超连接(HC)通过多路残差流和动态信息流提升表达能力。曼道约束(mHC)通过双随机残差混合稳定训练。但现有方法在生成器层面存在瓶颈:使用密集无结构生成器进行分支前聚合、残差混合与分支后重分配,导致参数量随流数急剧增长。为此,本文提出TEMPER——一种张量化高效曼道约束参数化方法,将生成器表示为输入流、特征和输出流三个维度的多维张量,并用张量网络参数化。该结构化低秩形式在保持逐标记曼道约束路由接口的同时显著降低参数增长。同时增强可解释性:i)张量秩控制学习路由子空间维度,全秩恢复稠密路由;ii)生成器近似误差界定了路由logits差异,进而约束路由块输出差异。大量实验表明,TEMPER在语言建模与常识推理任务中表现匹配或优于现有方法,且额外参数大幅减少。在8路残差流下,核心得分(CORE)最优,额外参数仅需mHC的约16%(即节省84%),展现出更强的性能-参数效率权衡。

原文摘要 · Abstract (English)

Residual connections rely on a static residual pathway, and are essential for training deep neural networks. Hyper-connections (HC) increase the expressivity of residual routing by incorporating multiple residual streams and learning dynamic information flow, while manifold-constrained (mHC) variants stabilize training through doubly stochastic residual mixing. However, a generator-level bottleneck remains in existing methods: they use dense, unstructured generators for pre-branch aggregation, residual mixing, and post-branch redistribution, which results in parameter count growing rapidly with the number of streams. To address this issue, we propose \underline{\textbf{T}}ensorized \underline{\textbf{E}}fficient \underline{\textbf{M}}anifold-constrained \underline{\textbf{P}}arameterization for \underline{\textbf{E}}xpressive Residual \underline{\textbf{R}}outing (\textbf{TEMPER}), which represents these generators as multi-way tensors over the input-stream, feature, and output-stream modes, and parameterizes them using tensor networks. Such a structured low-rank formulation is shown to preserve token-dependent manifold-constrained routing interface while substantially reducing parameter growth. It also promotes interpretability and intuition, as: i) tensor ranks control the dimensionality of the learned routing subspace, with full ranks recovering dense routing; while ii) the generator approximation errors bound differences in routing logits and, consequently, in the routed-block outputs. Comprehensive experiments show that TEMPER matches or outperforms existing methods across language modeling and commonsense reasoning tasks, while requiring substantially fewer additional parameters. At eight residual streams, TEMPER achieves the best CORE score while using about $84\%$ fewer additional parameters than mHC, thus showing a stronger performance-parameter efficiency trade-off.

神经网络路由机制张量网络参数效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。