arXiv:2510.10276cs.LGq-bio.NC2025-10被引 9

LLM中间信息丢失是训练中记忆需求的自然结果,非缺陷。

Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs

  • 用模拟人类记忆的任务训练模型,发现位置偏差源于不同记忆需求
  • 模型在长序列中两端表现更好,中间差,符合记忆的首因与近因效应
  • 该现象由训练任务、自回归结构和注意力聚集共同导致,适合研究模型机制者关注

大型语言模型(LLMs)在长上下文中的关键信息位于中间时性能下降,这种‘迷失于中间’现象类似于人类记忆的首因与近因效应。我们认为这并非单纯的信息丢失缺陷,而是预训练中不同信息检索需求的适应结果:部分任务要求对整个输入均匀回忆(长期记忆需求),另一些则优先最近信息(短期记忆需求)。通过在GPT-2和Llama变体上从零训练两个模拟人类记忆范式的简单任务,我们发现这种U型性能曲线确实会涌现。分析表明,近因效应直接对应短期记忆需求,而首因效应由均匀长期记忆需求引发,并受模型自回归特性及注意力聚集结构的影响。该发现也推广到更贴近预训练过程的序列补全任务。结果揭示了信息检索需求、模型架构与注意力动态如何共同导致LLM中的位置偏差。

原文摘要 · Abstract (English)

The performance of Large Language Models (LLMs) often degrades when crucial information is in the middle of a long context, a "lost-in-the-middle" phenomenon that mirrors the primacy and recency effects in human memory. We propose that this behavior is not simply a flaw indicative of information loss but an adaptation to different information retrieval demands during pre-training: some tasks require uniform recall across the entire input (a long-term memory demand), while others prioritize the most recent information (a short-term memory demand). Consistent with this view, we show that this U-shaped performance curve emerges when LLMs (GPT-2 and Llama variants) are trained from scratch on two simple human memory paradigms simulating long-term and short-term memory demands. Our analysis reveals that while the recency effect directly aligns with short-term memory demand in the training data, the primacy effect is induced by the uniform long-term memory demand and is additionally influenced by the model's autoregressive properties and the formation of attention sinks. Our main findings from simple human memory paradigms also generalize to a sequence completion task, which more closely resembles the next-token prediction process in LLM pre-training. Together, our findings reveal how information retrieval demands, model architecture, and structural attention dynamics during model training can jointly produce positional bias observed in LLMs.

大模型机制注意力机制记忆建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。