arXiv:2607.10441cs.LGcs.AI2026-07

让模型记忆只存独特信息,省下一半计算量还更可查。

Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams

论文配图:Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams
图 1 · 摘自论文原文
  • 按新信息触发缓存,内存大小随独特内容变化而非文本长度。
  • 在长上下文任务中,仅用一半 token 达到全注意力性能。
  • 记忆内容可查看,适合需要透明推理的医疗、金融场景。

上下文管理决定模型保留什么信息,现有方法以 token 为单位:将过去压缩成固定状态、为每个 token 保存键值对,或用窗口/淘汰规则设定预算。这些方法在冗余流中仍以 token 为单位,而任务实际依赖的是其中的独特信息。基于前作提出的仅当新键出现时才开启缓存的机制,我们构建了按新信息分配的缓存作为工作记忆组件,并根据任务需求组织上下文:需回忆的信息存入内容寻址的新颖性缓存,需摘要的信息存入递归状态,需局部性的信息存入时间窗口。实验证明,在字符级控制任务中,新颖性门控注意力仅处理约一半 token 即达全注意力性能;结合状态空间摘要后,成本更低却匹配全注意力效果,且优势随上下文变长而扩大;短而局部主导的序列仍宜用滑动窗口。在合成医保理赔数据的下一代码预测任务中,该组件优于全注意力和所有固定预算淘汰策略(千事件视野);而成本预测任务为摘要驱动,缓存无显著影响。留存记忆是可审查的模板、代码、药物或地点列表,而非黑箱状态。实验规模小,仅用公开数据,但证明上下文可按独特信息扩展而非 token 数量,且工作记忆具内容寻址与可审计性。

原文摘要 · Abstract (English)

Context engineering decides what information a model carries forward, and current designs meter it in tokens: compressing the past into a bounded recurrent state, keeping a key-value entry for every token, or imposing a fixed budget through a window or eviction rule. All three make the token the unit of memory even when the stream is redundant and the task depends on the distinct information it carries. Building on a companion mechanism paper that opens a cache slot only when an incoming key is novel, so memory scales with the number of distinct items rather than tokens, we develop that allocate-on-novelty cache as a working-memory component and organize context by how a task depends on the past: recall-carried information belongs in a content-addressed novelty cache, summary-carried information in a recurrent state, and locality-carried information in a recency window. The claim is empirical and bounded. On a matched character-level control, novelty-gated attention reaches full-attention performance while attending to about half the tokens, and coupling the cache with a state-space summary matches full-attention coupling at that reduced cost; the advantage grows as context lengthens, while a sliding window is preferable on short, locality-dominated spans. On next-code prediction over synthetic Medicare claims the coupled component leads full attention and every fixed-budget eviction policy at a thousand-event horizon, whereas cost forecasting over the same stream is summary-carried and the cache is neutral. The retained memory is an inspectable table of templates, codes, drugs, or places rather than an opaque state. The experiments are small-scale and use only public data; they establish the primitive that context can scale with distinct information rather than tokens, in a working memory that is content-addressable and auditable.

工作记忆可审计性上下文管理冗余流

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。