探究旋转位置编码对检索头的影响，发现其不阻止反而依赖于特定频率。
Does RoPE Prevent or Degrade Retrieval Heads? A Mechanistic Analysis Across Model Families
- 通过消融实验验证检索头是长序列回忆的必要组件。
- 不同θ值下检索头数量稳定，证明旋转编码未抑制其形成。
- 低频旋转编码维度对检索头功能有决定性影响，适用于模型优化者。
检索头作为长上下文记忆的机制基础，其与旋转位置编码（RoPE）的关系尚未明确。本文在四款7-8B开源模型（涵盖多头与分组查询注意力）及θ值跨度达100倍的条件下，通过配对种子针堆测试、层聚类置换和因果头掩码等方法进行分析。结果表明：(i) 检索头具有因果必要性——在OLMo-2中掩码87个检测到的检索头使召回率从1.00降至0.00，而掩码随机头无影响，该结果在Qwen中复现；(ii) 更高θ值并未减少检索头数量（LLaMA-3.1在θ=500K时有47个，而LLaMA-2在θ=10K时为42个），反驳了抑制假说；(iii) 规范-效用关系在不同模型家族中方向相反且显著（Qwen d=-0.49，OLMo d=+0.50，均显著；LLaMA无显著性），说明效应非θ驱动；(iv) 受Chiang and Yogatama (2025)启发，控制修补实验显示：将检索头的最低频RoPE维度置零会剂量式降低召回率（32/128维置零时从1.00降至0.18，随机维度仅0.98），且效果具头特异性和任务特异性。因果变量是RoPE频率而非规范-效用。该结论在五款模型（OLMo-2, Qwen2.5-7B/14B, Gemma-2, Mistral）中跨四个谱系和两种规模均成立。代码与配对种子测试工具已公开。
原文摘要 · Abstract (English)
Retrieval heads, attention heads that copy information from earlier context to the current position, have been proposed as the mechanistic substrate for long-context recall. Rotary position embeddings (RoPE) rotate queries and keys by frequencies decaying with a base hyperparameter theta, and a natural hypothesis is that this rotation either prevents retrieval heads from forming or degrades their function. We test both across four open-weight 7-8B models spanning multi-head and grouped-query attention and a 100x range of theta, using paired-seed needle-in-a-haystack tests, layer-clustered permutation, and causal head-masking. (i) Retrieval heads are causally necessary: masking the 87 detected heads in OLMo-2 collapses recall from 1.00 to 0.00, while masking matched random heads has no effect; this replicates in Qwen. (ii) Higher theta does not reduce retrieval-head count (LLaMA-3.1 at theta=500K has 47 heads vs LLaMA-2 at theta=10K with 42), refuting the prevention hypothesis. (iii) The norm-utility relation is family-specific and significant in opposite directions (Qwen d=-0.49, OLMo d=+0.50, both significant; LLaMA null); since OLMo and LLaMA-3.1 share theta=500K yet differ, the effect is not theta-driven. (iv) Building on Chiang and Yogatama (2025), a controlled patch shows that zeroing the lowest-frequency RoPE dimensions of retrieval heads degrades recall dose-dependently (1.00 to 0.18 when 32 of 128 dimensions are zeroed, vs 0.98 for random dimensions); the effect is head-specific and task-specific. The causal variable is RoPE frequency, not norm-utility. The direction holds in all five models patched (OLMo-2, Qwen2.5-7B/14B, Gemma-2, Mistral) across four lineages and two scales. We do not claim cross-model magnitude. Code and a paired-seed harness are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。