通过分层调节位置编码，解决大模型长文本中间信息丢失问题。
Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling
- 为每层分配不同缩放因子，减缓旋转位置编码的衰减速度。
- 在关键值检索数据集上平均准确率提升20%。
- 适合需要精准处理长文本中间内容的研究与应用。
尽管大语言模型在处理长上下文输入方面取得了显著进展，但仍面临‘中间信息丢失’问题，即上下文中间部分的关键信息常被弱化或遗忘。我们通过大量实验发现，这一问题可能源于旋转位置编码（RoPE）的快速长期衰减。为此，我们提出一种分层位置编码缩放方法，为每一层分配不同的缩放因子，以减缓RoPE引起的衰减，使模型更关注中间上下文。采用设计特殊的遗传算法结合贝塞尔曲线，高效搜索每层最优缩放因子，缩小搜索空间。实验证明，该方法显著缓解了‘中间信息丢失’问题，在关键值检索数据集上实现平均准确率提升20%。此外，相较于全层统一插值，分层插值结合PI与动态NTK位置编码方案，进一步增强了模型的外推能力。
原文摘要 · Abstract (English)
Although large language models (LLMs) have achieved significant progress in handling long-context inputs, they still suffer from the ``lost-in-the-middle'' problem, where crucial information in the middle of the context is often underrepresented or lost. Our extensive experiments reveal that this issue may arise from the rapid long-term decay in Rotary Position Embedding (RoPE). To address this problem, we propose a layer-specific positional encoding scaling method that assigns distinct scaling factors to each layer, slowing down the decay rate caused by RoPE to make the model pay more attention to the middle context. A specially designed genetic algorithm is employed to efficiently select the optimal scaling factors for each layer by incorporating Bezier curves to reduce the search space. Through comprehensive experimentation, we demonstrate that our method significantly alleviates the ``lost-in-the-middle'' problem. Our approach results in an average accuracy improvement of up to 20% on the Key-Value Retrieval dataset. Furthermore, we show that layer-specific interpolation, as opposed to uniform interpolation across all layers, enhances the model's extrapolation capabilities when combined with PI and Dynamic-NTK positional encoding schemes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。