提出轻量级残差路由机制，提升深度Transformer的训练效率与性能。
HAARES Half-Split Residual Basis Routing for Deep Transformers
- 采用半分割残差基路由，保留累积源并引入前后半段残差差值作为细节基。
- 48层模型下在三个数据集上均优于传统块级残差路由，且结果稳定可靠。
- 适合追求深度模型训练加速与收敛速度优化的研究者使用。
块级残差路由通过路由块摘要实现学习型残差聚合，但每个摘要将注意力与MLP更新的有序序列压缩为单一向量。本文提出 method{}，一种轻量级残差基路由器，保留累积块源，并增加一个半分割细节基，即前半段与后半段残差更新之差。该细节基进行RMS归一化并在线更新，暴露粗粒度块内轨迹信息，无需密集子层级路由。在OpenWebText、跨领域字符级基准及BPE分词的OpenWebText上，实验显示性能随深度变化：浅层增益小或不一致，48层模型中表现最稳定。201M参数48层设置下， method{} 在所有三组种子上均优于块注意力残差（Block AttnRes）；453M参数两组种子测试也呈现相同趋势。消融实验排除了源重复、随机符号细节、固定细节源偏置或块数变化的影响。成本分析表明，该方法算力开销低，但存在内存和路由开销；其相对计算成本随宽度增长而摊薄，且更早收敛可降低达到目标的时间。
原文摘要 · Abstract (English)
Block-level residual routing makes learned residual aggregation practical by routing over block summaries, but each summary compresses an ordered sequence of attention and MLP updates into one cumulative vector. We propose \method{}, a lightweight residual basis router that keeps the cumulative block source and adds one half-split detail basis, computed as the difference between first-half and second-half residual updates. The detail basis is RMS-matched and updated online, exposing coarse intra-block trajectory information without dense sublayer-level routing. Across OpenWebText, cross-domain character-level benchmarks, and BPE-tokenized OpenWebText, the empirical pattern is depth-dependent: gains are small or mixed at shallow depth and most reliable in 48-layer models. In the 201M 48-layer setting, \method{} improves over Block AttnRes across all three seeds, while a 453M two-seed probe shows the same direction. Ablations rule out source duplication, random signed details, fixed detail-source biases, or block-count changes alone. Cost analysis shows that the method is FLOP-light but not wall-clock-free: it adds memory and routing overhead, yet its relative arithmetic cost is amortized as width grows and earlier convergence can reduce time-to-target.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。