大模型难定位列表末尾项,影响代码理解与编辑。
The Position Curse: LLMs Struggle to Locate the Last Few Items in a List

- 用前后偏移定位法测试模型位置感知能力。
- 后向检索准确率远低于前向检索,尤其在末尾项上。
- 适合关注代码分析、模型索引能力的研究者。
现代大语言模型虽能从海量文本中精准定位单一信息,却难以准确识别短列表中的最后几项,我们称此为‘位置诅咒’。例如,在仅两行的代码片段中,Claude Opus 4.6 多次错误识别倒数第二行。为系统评估该缺陷,我们设计了两类互补任务:给定序列中某位置(以字母或词为单位),返回对应项;或给定一项,返回其位置。位置通过相对于起点、终点或列表内其他项的前向/后向偏移定义。在开源与前沿闭源模型中,后向检索性能显著落后于前向检索。为验证能否通过微调修复,我们构建了专用于位置学习的 PosBench 训练数据集。使用 LoRA 微调可提升前后向检索表现,并在保留的代码理解基准 PyIndex 上实现泛化,但绝对性能仍远未饱和。随着大模型作为编程代理处理大型代码库,精确索引能力将成为未来预训练目标与模型设计的关键。
原文摘要 · Abstract (English)
Modern large language models (LLMs) can find a needle in a haystack (locating a single relevant fact buried among hundreds of thousands of irrelevant tokens) with near-saturated accuracy, yet fail to retrieve the last few items in a short list. We call this failure the Position Curse. For instance, even in a two-line code snippet, Claude Opus 4.6 misidentifies the second-to-last line most of the time. To characterize this failure, we evaluated two complementary queries: given a position in a sequence (of letters or words), retrieve the corresponding item; and given an item, return its position. Each position is specified as a forward or backward offset from an anchor, either an endpoint of the list (its start or end) or another item in the list. Across both open-source and frontier closed-source models, backward retrieval substantially lags forward retrieval. To test whether this capability can be rescued by post-training, we constructed PosBench, a position-focused training dataset. LoRA fine-tuning improves both forward and backward retrieval and generalizes to a held-out code-understanding benchmark (PyIndex), yet absolute performance remains far from saturated. As LLM coding agents increasingly operate over large codebases where precise indexing becomes essential for code understanding and editing, position-based retrieval emerges as a key capability for future pretraining objectives and model design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。