arXiv:2609.07314cs.CLcs.LG2026-09

用内容地址取代位置编号,让模型更稳定处理长文本。

Content-Based Addressing for Long Context

  • 以内容生成单元地址,替代传统位置编码
  • 长文本下困惑度从12.09降至3.82,性能显著提升
  • 适合需要精准长程记忆的文本理解任务

旋转位置编码(RoPE)依赖词元的整数位置来决定注意力中的旋转,适用于局部顺序,但随着上下文增长会出现训练-测试位置不匹配:RoPE在未见过的偏移上产生相对旋转。现有方法通过缩放、插值、随机化或偏置位置来应对,但仍依赖不断增长的词元计数器获取位置信息。本文提出将词元流划分为若干单元,每个单元内保留普通RoPE位置,而为已完成的单元分配基于其内容计算出的地址。新增单元时,直接应用已学习的映射到新内容,而非扩展位置范围或标识表。我们证明该结构能精确保持局部RoPE特性,在插入或重排其他单元时,两个固定词元间的注意力对比不变,且仅因增加单元数而产生新相对旋转。在字符级Tiny Shakespeare诊断中,模型在256字符训练后,验证困惑度在256字符时为4.04,4096字符时为3.82;而连续RoPE从4.71升至12.09。另一实验显示,内容地址可有效检索并利用多个序列化事实。这些是受控的浅层实验,非大规模基准,但支持明确建议:局部用位置定位,跨单元用内容定位。

原文摘要 · Abstract (English)

Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention. This works well for local token order, but increasing context length creates a positional train-test mismatch: RoPE produces relative rotations at offsets not seen during training. Methods that rescale, interpolate, randomize, or bias positions specify how attention handles those offsets, but still derive positional information from a growing token counter. We instead divide a token stream into units, retain ordinary RoPE positions within each unit, and assign every completed unit an address computed from its content. Adding units then applies the same learned map to new content rather than extending a positional range or an identifier table. We prove that this construction preserves local RoPE exactly, leaves the attention comparison between two fixed tokens unchanged when other units are inserted or reordered, and does not create new relative rotations merely because more units are added. In a character-level Tiny Shakespeare diagnostic, a model trained on 256-character contexts has validation perplexity 4.04 at 256 characters and 3.82 at 4096, while continuous RoPE changes from 4.71 to 12.09. A second diagnostic shows that content-based addressing can retrieve and use information from multiple serialized facts. These are controlled shallow experiments, not scale benchmarks, but they support a direct prescription: use position to address locally and content to address across units.

位置编码长文本注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。