arXiv:2604.09670cs.LGcs.AI2026-04

大模型像人一样有记忆局限,因信息干扰导致记不住太多内容。

In-context superposition: human-like working memory interference in large language models

  • 用上下文叠加编码实现多信息共存,但引发相互干扰。
  • 记忆负载越高,准确率越低,且受最近信息影响更大。
  • 发现可干预干扰项提升表现,适合研究认知机制的人看。

智能系统需在线维护和操作任务相关信息以适应动态环境,这一能力称为工作记忆,是人类推理的基础。然而,人类工作记忆极为有限,仅能维持三到四个信息项,尽管大脑拥有数十亿神经元。令人惊讶的是,大型语言模型(LLMs)虽具备不同底层结构并可通过注意力直接访问过往上下文,却也表现出类似的限制。我们提出,这种限制反映了共享表征的普遍权衡:表征压缩与复用支持高效学习与泛化,但也导致同时激活的表征产生干扰。我们发现,一个两层变压器在工作记忆任务上可完美解决,但多样化的训练过的LLM表现出类人限制:性能随记忆负荷下降,且检索受近期性和刺激统计偏倚影响。与人类一致,LLM的工作记忆表现也与整体模型能力相关。机制上,我们表明LLM以纠缠表示编码多个记忆——我们称之为‘上下文超位置’,并在读取时逐步抑制竞争内容并对齐目标。此外,因果干预抑制干扰信息可提升性能。这些发现表明,工作记忆容量反映在干扰下选择任务相关信息的能力,这是生物与人工系统共同面临的计算挑战。

原文摘要 · Abstract (English)

Intelligent systems must maintain and manipulate task-relevant information online to adapt to dynamic environments. This capacity, known as working memory, is fundamental to human reasoning. Yet, human working memory is strikingly limited, maintaining only three to four items in a brain with billions of neurons. Surprisingly, large language models (LLMs), despite different substrates and direct access to prior context through attention, exhibit similar working memory limitations. Why should such different systems face analogous constraints? We propose that working memory limitations reflect a general trade-off of shared representations: representational compression and reuse support efficient learning and generalization, but also cause simultaneously active representations to interfere. We show a two-layer transformer trained on a working memory task can solve it perfectly, but diverse trained LLMs exhibit human-like limitations: performance declines with memory load, while retrieval is biased by recency and stimulus statistics. Mirroring humans, working memory performance in LLMs is also associated with broader model capability. Mechanistically, we show that LLMs encode multiple memories in entangled representations --- a condition we call \emph{in-context superposition} --- and progressively suppress competing content while aligning the target with the readout. Moreover, a causal intervention that suppresses interfering information improves performance. Together, these findings suggest that working memory capacity reflects the ability to select task-relevant information under interference, a computational challenge shared by biological and artificial systems.

工作记忆大模型干扰机制认知建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。