让注意力值路径感知位置，提升长距离依赖建模能力。
RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways
- 值向量随键向量同步旋转，实现位置敏感的值传递
- 在少样本上下文学习等任务上显著优于传统RoPE
- 无需额外参数，适用于视觉、机器人和大模型场景
旋转位置编码（RoPE）使注意力得分具有位置相关性，但值路径仍对位置不敏感：无论值标记与查询的距离如何，其传递的信息相同。我们提出RoVE，一种无参数的改进方法，通过将值向量与键向量同步旋转，使值路径具备位置敏感性，并证明该设计将RoPE注意力转化为注意力卷积。这一新视角统一了计算机视觉、机器人学及现代大模型架构中多个独立提出的相同操作。训练124M和354M的GPT-2模型表明，在少样本上下文学习、分布外困惑度和长上下文检索任务上，RoVE持续优于RoPE，尤其在需要长程聚合的任务中优势明显。
原文摘要 · Abstract (English)
Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway position-blind: the message sent by a value token is the same regardless of its distance from the query. We propose RoVE, a parameter-free modification that makes values position-sensitive by rotating them simultaneously with keys, and show that it turns RoPE attention into attentive convolution. This new perspective unifies several independent formulations of the same operation across computer vision, robotics, and modern LLM architectures. Trained 124M and 354M GPT-2 models show consistent empirical gains over RoPE on few-shot in-context learning, out-of-distribution perplexity, and long-context retrieval, with the clearest improvements on tasks that require long-range aggregation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。