让残差连接学会自我判断，大模型效果更优
Review Residuals: Update-Conditioned Residual Gating for Transformers
- 用输入相关的门控机制动态调节残差更新强度
- 590M以上模型显著优于标准残差和高速门控结构
- 优势随模型规模增大而增强，适合大模型训练
残差连接始终以固定系数1累加子层更新，不评估更新可靠性。受人类因素中独立验证原则启发，我们提出Review Residuals，通过一个由当前状态和待更新值共同决定的可学习门控系数来缩放每一步更新：h_l = h_{l-1} + r_l * u_l，其中r_l = sigmoid(W[RMSNorm(h_{l-1}), RMSNorm(u_l)]）。与以往门控残差不同，该门控依赖于更新本身。我们报告两项发现：第一，深度稳定性结果表明，采用凸型（Highway式）门控会引发梯度消失，训练无法超过约20层；而加性、保持恒等性的形式可在所有测试深度下稳定训练。第二，规模涌现结果表明，在五个不同规模（60M-1B参数，多种子）从头训练中，小模型无优势，但在590M时显著优于参数匹配的Highway门控和标准残差（p<0.05），1B模型优势更大，且收益随模型规模增长而非衰减。
原文摘要 · Abstract (English)
Residual connections add every sublayer's proposed update with a fixed coefficient of one; the network never evaluates whether an update is reliable before committing it. Drawing on the human-factors principle of independent verification, we introduce Review Residuals, which scale each update by a learned, input-dependent gate conditioned on both the current state and the proposed update: h_l = h_{l-1} + r_l * u_l with r_l = sigmoid(W[RMSNorm(h_{l-1}), RMSNorm(u_l)]). Conditioning the gate on the update is the property that distinguishes it from prior gated and scaled residuals. We report two findings. First, a depth-stability result: a convex (Highway-style) form of the gate reintroduces vanishing gradients and fails to train beyond ~20 layers, whereas the additive, identity-preserving form trains stably at all depths we tested. Second, an emergence-with-scale result: trained from scratch across five sizes (60M-1B parameters, multi-seed), Review Residuals show no advantage at small scale but at 590M significantly outperform both a parameter-matched Highway gate and a parameter-matched standard residual (p<0.05), with a larger advantage at 1B. The benefit grows with model size rather than shrinking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。