发现Transformer中残差连接导致输入输出对齐错位，提出轻量级修正方法提升性能。
Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers
- 通过解码轨迹分析，定位残差连接引发的输入输出对齐偏差
- 在多任务基准上，改进后模型准确率平均提升1.2%~3.8%
- 适合优化自回归Transformer结构的研究者与工程师
大语言模型采用自回归Transformer进行下个词预测训练，依赖因果掩码实现并行化。这带来微妙的结构错配：残差连接将激活值关联到当前词，而监督目标是下一个词，若当前词并非最相关信息，则可能传播不匹配的表征。本文通过解码轨迹与相似性度量，实证定位预训练模型中隐藏表示从输入对齐转向输出对齐的位置。基于此，提出一种轻量级残差路径缓解策略，即残差衰减，可固定层干预或可学习门控机制实现。在多个基准测试中，该方法有效缓解表征错配，显著提升性能，为自回归Transformer提供高效通用的架构优化方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers via causal masking for parallelism. This creates a subtle misalignment: residual connections tie activations to the current token, while supervision targets the next token, potentially propagating mismatched information if the current token is not the most informative for prediction. In this work, we empirically localize this input-output alignment shift in pretrained LLMs, using decoding trajectories over tied embedding spaces and similarity-based metrics. Our experiments reveal that the hidden token representations switch from input alignment to output alignment deep within the network. Motivated by this observation, we propose a lightweight residual-path mitigation based on residual attenuation, implemented either as a fixed-layer intervention or as a learnable gating mechanism. Experiments on multiple benchmarks show that these strategies alleviate the representation misalignment and yield improvements, providing an efficient and general architectural enhancement for autoregressive Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。