改进位置编码，让大模型更准确复制长文本。
Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
- 将文本转为二维网格，用行列坐标替代一维位置编码。
- 在数百倍于训练长度的输入上实现完美复制，传统方法大幅落后。
- 适合需要精准文本复制的任务，如代码生成、文档处理。
尽管大语言模型可在数秒内解决复杂推理问题，我们发现即使是前沿模型也无法完成一个看似简单的任务：在上下文窗口内精确复制输入字符串。我们将其归因于Transformer架构中的位置编码，其归纳偏置倾向于通过匹配局部上下文的捷径来复制，而非精确定位对应位置。为此，我们提出2D-RoPE，将文本组织为二维网格，为每个标记分配行号和列号。在此视角下，复制变为固定列偏移的检索，学习难度显著降低。在合成复制实验中，使用2D-RoPE的浅层Transformer在输入长度远超训练长度数百倍时仍能实现完美复制，而标准位置编码表现远差。进一步在DCLM数据集上进行大规模预训练（最大1.4B参数）也证实了2D-RoPE在复制任务上的持续优势。结果表明，以二维方式看待文本可提升语言建模性能，鼓励未来探索二维位置编码潜力。
原文摘要 · Abstract (English)
While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an input string that lies well within their context windows. We attribute this failure to positional encodings in Transformer architectures, whose inductive bias favors copying through a shortcut based on matching local contexts rather than carefully locating the corresponding input positions. To address this issue, we introduce 2D-RoPE, which organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID. Under this view, copying becomes simply retrieving input tokens at a fixed column offset, which makes the task easy to learn. In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training, whereas standard positional encodings fall far behind. We further show that the advantage of 2D-RoPE language models on copy tasks consistently holds in large-scale pretraining on DCLM with model sizes up to 1.4B parameters. Overall, our results suggest that viewing text in 2D can benefit language modeling, and we hope this encourages future work to further explore the potential of 2D positional encodings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。