通过分层位置嵌入缩放，解决长文本中信息丢失问题。
Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling
- 为每层分配不同缩放因子，动态平衡注意力分布。
- 在键值检索任务上提升11.2%准确率，无需微调或增加延迟。
- 用遗传算法结合贝塞尔曲线高效搜索最优参数，适合长文本场景。
大语言模型仍面临长文本中的“中间信息丢失”问题，即关键信息在输入中间部分被弱化或忽略。现有方法虽尝试通过多尺度旋转位置编码（RoPE）缓解，但通常带来高延迟或依赖次优的手动缩放策略。为此，我们提出分层位置嵌入缩放（LPES）方法，为每一层分配独立的缩放因子，在不微调参数且不增加推理延迟的前提下实现更均衡的注意力分布。采用特制遗传算法结合贝塞尔曲线显著缩小搜索空间，高效选取各层最优缩放因子。大量实验表明，LPES有效缓解位置注意力偏差，在多个长上下文基准上持续提升性能，键值检索任务最高提升11.2%准确率。
原文摘要 · Abstract (English)
Large Language Models (LLMs) still struggle with the ``lost-in-the-middle'' problem, where critical information located in the middle of long-context inputs is often underrepresented or lost. While existing methods attempt to address this by combining multi-scale rotary position embeddings (RoPE), they typically suffer from high latency or rely on suboptimal hand-crafted scaling strategies. To overcome these limitations, we introduce a layer-specific positional embedding scaling~(LPES) method that assigns distinct scaling factors to each layer. LPES achieves a more balanced attention distribution without fine-tuning model parameters or increasing inference delay. A specially designed genetic algorithm is employed to efficiently select the optimal scaling factors for each layer by incorporating Bézier curves to significantly reduce the search space. Extensive experiments demonstrate that LPES effectively mitigates positional attention bias and delivers consistent improvements across multiple long-context benchmarks, yielding up to an $11.2$\% accuracy gain on the key-value retrieval dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。