arXiv:2601.12906cs.CL2026-01ACL被引 7

让大模型更高效地记住长文本关键信息,减少无效计算。

Gated Differentiable Working Memory for Long-Context Language Modeling

  • 用可学习的门控机制决定哪些内容该记入临时记忆
  • 仅用四分之一的计算量就达到甚至超越现有方法效果
  • 适合需要快速适应新语境的长文本任务

长上下文挑战变换器:注意力分数在数千个词元间稀释,中间关键信息常被丢失,模型在推理时难以适应新模式。近期测试时自适应方法通过维护临时参数(工作记忆)缓解此问题,但现有方法采用统一写入策略,在低价值区域浪费计算,且在语义异质上下文中梯度方差高。本文将测试时自适应重定义为预算约束下的记忆巩固问题,关注在有限计算下应将哪些上下文部分纳入工作记忆。提出 Gdwm(门控可微工作记忆)框架,引入写入控制器,基于上下文效用(一种信息论度量的长程依赖性)估算并分配梯度步数,同时保持全局覆盖。在 ZeroSCROLLS 与 LongBench v2 上的实验表明,Gdwm 在仅需均匀基线 1/4 梯度步数的情况下,性能相当或更优,确立了测试时自适应的新效率-性能帕累托前沿。

原文摘要 · Abstract (English)

Long contexts challenge transformers: attention scores dilute across thousands of tokens, critical information is often lost in the middle, and models struggle to adapt to novel patterns at inference time. Recent work on test-time adaptation addresses this by maintaining a form of working memory -- transient parameters updated on the current context -- but existing approaches rely on uniform write policies that waste computation on low-utility regions and suffer from high gradient variance across semantically heterogeneous contexts. In this work, we reframe test-time adaptation as a budget-constrained memory consolidation problem, focusing on which parts of the context should be consolidated into working memory under limited computation. We propose Gdwm (Gated Differentiable Working Memory), a framework that introduces a write controller to gate the consolidation process. The controller estimates Contextual Utility, an information-theoretic measure of long-range contextual dependence, and allocates gradient steps accordingly while maintaining global coverage. Experiments on ZeroSCROLLS and LongBench v2 demonstrate that Gdwm achieves comparable or superior performance with 4$\times$ fewer gradient steps than uniform baselines, establishing a new efficiency-performance Pareto frontier for test-time adaptation.

长文本建模工作记忆测试时自适应高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。