让Transformer实现终身学习，需用参数化注意力机制。
Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention
- 用可参数化的注意力替代传统softmax，动态建模长期上下文关系。
- 仅需固定内存，就能处理无限长序列，突破传统注意力的二次复杂度瓶颈。
- 适合研究长期智能体、高效推理的学者，推动硬件受限下的持续学习发展。
终身持续学习仍是实现类人智能的主要障碍。现代Transformer在上下文学习中展现出初步智能火花，但其注意力机制的二次复杂度限制了在任意长序列上的应用。本文主张，将上下文学习扩展至终身学习场景是解决人工智能代理持续学习的有效路径。关键在于引入参数化注意力机制，使Transformer能在固定硬件预算下理解一生的上下文信息。此类机制在测试时通过参数化回归学习键与值之间的关系，取代不断增长的键值缓存，保持恒定内存开销。我们对参数化方法（线性注意力、状态空间模型、快速权重编程、测试时训练层）与非参数方法（如softmax注意力）进行了对比分析。当前参数化注意力仍因记忆容量有限或在线更新成本高而难以支持真正的终身学习。为此，本文提出一系列开放问题与新洞见，以引导该领域向长周期智能体方向发展。
原文摘要 · Abstract (English)
Lifelong continual learning remains an obstacle on the path to human-like intelligence. Modern transformers show sparks of intelligence with in-context learning. The quadratic nature of attention, however, prohibits transformers from performing this process on arbitrarily long sequences. In this work, we argue that extending in-context learning to lifelong settings is a practical solution for continual learning in AI agents. In particular, we argue that \emph{parametric forms of attention} are needed to understand a lifetime of context with transformers on a fixed hardware budget. These attention mechanisms learn the relationship between keys and their associated values at test-time with parametric regression. Our generalization of parametric approaches (linear attention, state-space models, fast weight programmers, and test-time training layers) contrasts with nonparametric counterparts like softmax attention. They replace the ever-growing key-value cache with an online-trainable neural network, maintaining a constant memory footprint. We highlight how parametric attention currently fall short of lifelong learning due to limited memory capacity or costly online updates. To address these issues, we pose a set of open questions with novel insights to guide the field toward long-horizon agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。