让旋转位置编码的频率可学习，提升语言模型性能。
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
- 将旋转位置编码的频率设为可学习参数，取代固定超参。
- 在52M至2.5B参数模型上，均优于传统RoPE和部分RoPE。
- 大模型下性能更优，相同效果需3.4%更多算力才可达成RoPE水平。
旋转位置编码（RoPE）是当前主流语言模型中广泛使用的相对位置编码方式。它通过旋转查询与键向量的二维块，基于相对位置偏移进行操作。传统RoPE的旋转频率通常由固定基数超参数决定，遵循几何序列。先前工作通过增大该参数以减缓旋转，或仅对部分QK维度应用RoPE来提升性能。本文提出学习型旋转位置编码（LeRoPE），将每个频率设为可学习标量，而非固定超参数。我们在从52M到2.5B参数的多尺度语言模型上训练并验证了该方法。实验发现并分析了一种高范数、具位置特性的LeRoPE频带。结果表明，LeRoPE在所有规模下持续优于标准RoPE和部分RoPE，且在最大规模下，罗佩需额外3.4%的计算量（FLOPs）才能达到LeRoPE性能。
原文摘要 · Abstract (English)
Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models. RoPE rotates two-dimensional chunks of query and key vectors, operating as a function of their relative positional offset. The position-wise rates of rotation in RoPE typically follow a geometric sequence specified by a fixed base-frequency hyperparameter. Prior work has improved performance by either increasing this parameter to slow rotation or by applying RoPE to only a subset of QK dimensions. In this work we modify RoPE by learning a scalar per frequency, treating frequencies as learnable parameters rather than hyperparameters. We validate Learned RoPE by training a ladder of language models from scratch, ranging from 52M to 2.5B parameters. We observe and analyze the emergence of a high-norm, positional LeRoPE band. LeRoPE consistently outperforms RoPE and partial RoPE across all scales, with RoPE requiring 3.4% more compute (FLOPs) to match LeRoPE at the largest scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。