数据决定RoPE频率使用,解释了模型为何偏好特定频率。
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

- 用数据结构匹配解释频率选择,频率与相对距离分布匹配
- 自然语言具多尺度依赖,导致模型偏爱中低频段
- 频率缩放可提升长文本泛化,前提是依赖结构自相似
旋转位置编码(RoPE)为Transformer提供固定的位置频率网格,但训练后的模型对这些频率的使用极不均匀。本文从数据出发,提出频率选择是为了匹配训练数据的相对距离结构。将每个频率视为一种位置视角,我们形式化了场域分辨率权衡:对于宽度为 $W$ 的数据依赖分布,最优频率约为 $1/W$。该频率匹配原理解释了合成数据和文本数据中的控制性观察,并表明语言模型中常见的中低频带源于自然语言的多尺度依赖结构。进一步发现,频率选择与基于位置插值的长度外推相关:降低频率可扩大有效视野但降低分辨率。当长上下文依赖是训练时依赖的近似缩放时此法有效,但若依赖不随上下文长度缩放则失效。实证表明,自然语言在不同位置尺度上具有近似自相似性,这解释了测试时频率缩放为何能支持长上下文泛化。整体而言,本文揭示了RoPE频率使用的数据驱动机制,并指出长上下文泛化依赖两种尺度匹配:训练时频率与依赖结构的匹配,以及频率缩放与依赖扩展方式的匹配。
原文摘要 · Abstract (English)
Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what determines this frequency usage and propose a data-centered explanation: RoPE frequencies are selected to match the relative-distance structure of the training data. Viewing each frequency as a positional lens, we formalize a field-resolution tradeoff and show that, for a data-induced dependency profile of width $W$, the optimal frequency scales as $1/W$. This frequency-matching principle explains controlled observations on synthetic and text-based data, and suggests that the mid-low frequency bands observed in language models arise from the multi-scale dependency structure of natural language. We further connect frequency selection to position-interpolation-based length generalization: scaling frequencies down expands the effective field while reducing resolution. This helps when longer-context dependencies are approximate dilations of those seen during training, but can fail when relevant dependencies do not scale with context length. Empirically, we show that natural language exhibits approximate self-similarity across positional scales, explaining why test-time frequency scaling can support long-context generalization. Overall, our results identify a data-driven mechanism behind emergent RoPE frequency usage and show that long-context generalization depends on two forms of scale matching: between learned frequencies and training-time dependencies, and between frequency scaling and how those dependencies extend to longer contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。