发现注意力模型中敏感层与位置编码层不重合，却仍能提升性能。
Sensitivity-Positional Co-Localization in GQA Transformers
- 通过新指标定位任务敏感层，仅在这些层用LoRA微调。
- 早期层对位置编码最敏感，晚期层对任务正确性最敏感。
- 在多个评测中表现优于其他配置，适合资源受限的模型优化。
我们研究了分组查询注意力（GQA）Transformer中的一个基础结构问题：对任务正确性最敏感的层是否与位置编码调整最具影响力的层一致？我们称此为共定位假说，并在拥有32层、4:1查询-键值头比的Llama 3.1 8B模型上进行了验证。提出 extsc{LSLORA}，将LoRA适配限制在通过新提出的正确性差异隐藏状态度量识别出的层；以及GARFA（GQA感知的RoPE频率适配），在每层为每个键值头添加8个可学习标量乘子。结果与共定位假说相反：任务敏感层集中在网络后段（ℓ∈{23–31}），而RoPE影响显著层集中在前段（ℓ∈{0–9}），斯皮尔曼相关系数rs = -0.735（p = 1.66×10⁻⁶）。尽管存在反向共定位，四路跨层消融实验显示，将两项干预同时应用于敏感层，在六个不同基准测试（MMLU、GPQA、HumanEval+、MATH、MGSM、ARC）上均优于所有其他配置，性能提升4–16个百分点，且在总计算成本为$100时接近Claude 3.5 Haiku在HumanEval+上的表现（67.1% vs. 68.3%）。
原文摘要 · Abstract (English)
We investigate a fundamental structural question in Grouped Query Attention (GQA) transformers: do the layers most sensitive to task correctness coincide with the layers where positional encoding adaptation has the greatest leverage? We term this the co-localization hypothesis and test it on Llama 3.1 8B, a 32-layer GQA model with a 4:1 query-to-key-value head ratio. We introduce \LSLORA, which restricts LoRA adaptation to layers identified via a novel correctness-differential hidden-state metric, and GARFA (GQA-Aware RoPE Frequency Adaptation), which attaches 8 learnable per-KV-head scalar multipliers to each targeted layer. Contrary to the co-localization hypothesis, we discover strong anti-localization: task-sensitive layers concentrate in the late network ($\ell\in\{23\text{-}31\}$) while RoPE-influential layers dominate the early network ($\ell\in\{0\text{-}9\}$), yielding Spearman $r_s = -0.735$ ($p = 1.66\times10^{-6}$). Despite this anti-localization, a 4-way cross-layer ablation shows that applying both interventions to the sensitivity-identified layers outperforms all alternative configurations by 4-16 percentage points across six diverse benchmarks (MMLU, GPQA, HumanEval+, MATH, MGSM, ARC), approaching Claude 3.5 Haiku on HumanEval+ (67.1% vs. 68.3%) at \$100 total compute cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。