改进Transformer记忆能力，让模型更高效利用历史信息。
You Do Not Fully Utilize Transformer's Representation Capacity
- 引入层内记忆机制，融合多层表示增强上下文感知。
- 在语言建模和合成推理任务中提升每计算量的困惑度表现。
- 揭示了局部与长距离特征的系统性复用，适合深度模型研究者。
与将历史压缩为单一隐藏状态的RNN不同，Transformer可直接关注所有过去标记。然而标准Transformer仅依赖前一层的隐藏状态表示完整上下文，这种设计导致表征坍塌压力并降低性能。为此，我们提出轻量级扩展层内记忆（LIMe），利用现有键值缓冲区，并学习每头、每层的路由权重，以整合前层表示。在语言建模、合成推理及深层架构中，LIMe在研究范围内提升了每FLOP的困惑度，显著改善合成任务表现，同时保持更高值向量熵与标记可分性。学习到的路由权重揭示了局部与长距离特征的系统性复用，表明LIMe在不增加隐藏状态尺寸的前提下丰富了注意力时间记忆。代码已开源：https://github.com/corl-team/lime。
原文摘要 · Abstract (English)
In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standard Transformers rely solely on the hidden state from the previous layer to represent the entire context. We show that this design creates pressure toward representation collapse and can degrade performance. To address this issue, we introduce Layer-Integrated Memory (LIMe), a lightweight extension that leverages existing key-value buffers and learns per-head, per-layer routing weights to integrate representations from previous layers. Across language modeling, synthetic reasoning, and deep architectures, LIMe improves perplexity per FLOP in the studied regimes and yields strong gains on synthetic tasks while preserving higher value-vector entropy and token separability. Finally, learned routing weights reveal systematic reuse of local and long-distance features, showing how LIMe enriches attention-time memory without increasing hidden-state size. Code is available at https://github.com/corl-team/lime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。