RoPE在长序列中旋转角度过大，导致部分维度失效。
The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval
- 通过控制实验发现RoPE使某些维度利用率降低。
- 三款大模型验证：这些失效维度无助于长距离问答。
- 适合关注位置编码效率的模型优化研究者。
旋转位置编码（RoPE）被广泛应用于大型语言模型（LLM）的注意力头中。它根据输入序列中位置的不同，对查询和键向量的各个维度施加不同的旋转角度。在长上下文建模中，位置范围可能极大，导致某些维度被旋转了极大幅度的角度。我们假设这种大范围的旋转可能阻碍模型有效利用这些维度。为验证该假设，我们进行了受控实验，结果显示应用RoPE确实降低了某些维度的使用效率。对三款大型语言模型的分析进一步表明，这些低效维度在长上下文问答任务中未提供显著帮助。
原文摘要 · Abstract (English)
The Rotary Position Embedding (RoPE) is widely used in the attention heads of many large language models (LLM). It rotates dimensions in the query and the key vectors by different angles according to their positions in the input sequence. For long context modeling, the range of positions may vary a lot, and thus RoPE rotates some dimensions by a great range of angles. We hypothesize that the wide range of rotation angles may prevent LLMs from utilizing those dimensions. To validate this hypothesis, we present a controlled experiment showing that applying RoPE causes low utility of certain dimensions. Our analyses on three LLMs also indicate that these dimensions do not help LLMs do long-context question answering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。