给Transformer加生物启发的快速记忆机制，让模型更快适应新任务。
Enabling Robust In-Context Memory and Rapid Task Adaptation in Transformers with Hebbian and Gradient-Based Plasticity
- 用海布学习或梯度更新方式动态调整权重，实现快速序列适应。
- 海布规则在少样本任务中损失更低，长期任务则梯度方法更优。
- 适合研究快速适应、类脑计算或提升模型泛化能力的场景。
大型语言模型在推理时依赖静态权重，而生物系统通过突触可塑性持续适应。本文探究是否可通过显式的、仿生的可塑性机制，使Transformer实现更快的序列内适应。为此，我们在仅解码器的Transformer中引入两种快速权重模块：(i) 神经调制海布规则，或 (ii) Duan等（2023）提出的基于梯度的可塑性机制。在复制、回归及少样本分类任务（CIFAR-FS、Omniglot）上，海布可塑性始终表现更优，损失更低且泛化更强；而梯度方法在长程信用分配任务中更佳。当关联关系短且线性可分时，静态权重已足够，明确了可塑性生效的边界条件。对调制信号的分析显示，梯度规则维持大而持久的更新，而海布机制则在显著事件处被锐利激活。结果表明，显式可塑性可补充注意力机制，实现快速、任务特定的适应，并厘清不同机制的适用场景。
原文摘要 · Abstract (English)
Large language models display in-context learning as an emergent effect of scale, but they rely on static weights during inference. In contrast, biological systems continually adapt via synaptic plasticity. We investigate whether explicit, biologically inspired plasticity can endow Transformers with faster in-sequence adaptation. To this end, we augment decoder-only Transformers with fast-weight modules updated either by (i) a neuromodulated Hebbian rule or (ii) the gradient-based plasticity mechanism of Duan et al. (2023). Across copying, regression, and few-shot classification tasks (CIFAR-FS, Omniglot), Hebbian plasticity consistently achieves lower loss and stronger few-shot generalization, while gradient-based updates perform best on long-horizon credit assignment. When associations are short and linearly separable, static weights suffice, defining a clear boundary condition for when plasticity helps. Analysis of learned modulatory signals reveals that gradient-based rules maintain large, persistent updates, whereas Hebbian plasticity is sharply gated around salient events. Together, these results show that explicit plasticity complements attention by enabling rapid, task-specific adaptation, and clarify when different plasticity mechanisms are most effective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。