提出新型位置编码PoPE，分离内容与位置信息，提升模型性能。
Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
- 用极坐标分解位置编码，解耦内容与位置的混淆
- 在音乐、基因组和语言建模中降低困惑度，提升任务表现
- 零样本长序列外推能力强，优于RoPE和需微调的YaRN
Transformer中的注意力机制同时依赖内容（what）和序列位置（where）。我们分析发现，主流的RoPE旋转位置编码会混淆这两者，尤其在需要独立匹配内容或位置的任务中影响性能。为此提出极坐标位置编码（PoPE），消除what-where耦合。PoPE在仅依赖位置或内容的诊断任务中表现显著更优。在音乐、基因组和自然语言的自回归序列建模中，使用PoPE的Transformer在评估损失（困惑度）和下游任务性能上均优于基于RoPE的基线。语言建模中的优势在124M至774M参数规模下均持续存在。关键的是，PoPE展现出强零样本长度外推能力，不仅优于RoPE，也优于需额外微调和频率插值的外推专用方法YaRN。
原文摘要 · Abstract (English)
The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where. We present an analysis indicating that what and where are entangled in the popular RoPE rotary position embedding. This entanglement can impair performance particularly when decisions require independent matches on these two factors. We propose an improvement to RoPE, which we call Polar Coordinate Position Embeddings or PoPE, that eliminates the what-where confound. PoPE is far superior on a diagnostic task requiring indexing solely by position or by content. On autoregressive sequence modeling in music, genomic, and natural language domains, Transformers using PoPE as the positional encoding scheme outperform baselines using RoPE with respect to evaluation loss (perplexity) and downstream task performance. On language modeling, these gains persist across model scale, from 124M to 774M parameters. Crucially, PoPE shows strong zero-shot length extrapolation capabilities compared not only to RoPE but even a method designed for extrapolation, YaRN, which requires additional fine tuning and frequency interpolation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。