提出Palimpsa模型，让注意力机制像人一样学会记忆、学习和遗忘。
Learning to Remember, Learn, and Forget in Attention-Based Models
- 将上下文学习视为持续学习问题，用贝叶斯元可塑性控制注意力状态的更新强度。
- 在多查询关联回忆任务上超越基线模型，长序列记忆能力显著提升。
- 适用于需要长期记忆与抗干扰的复杂序列建模任务，如常识推理。
Transformer中的上下文学习（ICL）充当在线联想记忆，支撑其在复杂序列任务上的高性能。然而，在门控线性注意力模型中，该记忆容量固定且易受干扰，尤其在长序列下表现不佳。本文提出Palimpsa，将ICL视为需解决稳定-可塑性困境的持续学习问题。Palimpsa采用贝叶斯元可塑性，使每个注意力状态的可塑性由一个基于先验分布的重要性状态决定，以捕获累积知识。我们证明多种门控线性注意力模型是特定架构选择与后验近似的结果，而Mamba2是其中遗忘主导的特例。这一理论联系使得任意非元可塑模型可转化为元可塑模型，显著扩展记忆容量。实验表明，Palimpsa在多查询关联回忆（MQAR）基准和常识推理任务上均持续优于基线模型。
原文摘要 · Abstract (English)
In-Context Learning (ICL) in transformers acts as an online associative memory and is believed to underpin their high performance on complex sequence processing tasks. However, in gated linear attention models, this memory has a fixed capacity and is prone to interference, especially for long sequences. We propose Palimpsa, a self-attention model that views ICL as a continual learning problem that must address a stability-plasticity dilemma. Palimpsa uses Bayesian metaplasticity, where the plasticity of each attention state is tied to an importance state grounded by a prior distribution that captures accumulated knowledge. We demonstrate that various gated linear attention models emerge as specific architecture choices and posterior approximations, and that Mamba2 is a special case of Palimpsa where forgetting dominates. This theoretical link enables the transformation of any non-metaplastic model into a metaplastic one, significantly expanding its memory capacity. Our experiments show that Palimpsa consistently outperforms baselines on the Multi-Query Associative Recall (MQAR) benchmark and on Commonsense Reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。