给视觉Transformer加快速权重,让模型在少量样本下更快适应。
Where to Bind Matters: Hebbian Fast Weights in Vision Transformers for Few-Shot Character Recognition
- 在ViT、DeiT、Swin等模型最后阶段加入快速权重模块,实现快速适应。
- 在Omniglot上1次学习准确率达96.2%,5次学习达99.2%,优于基线0.3个百分点。
- 揭示了窗口偏置与快速绑定的协同作用,适合低数据场景下的快速学习研究。
标准Transformer在训练中学习固定慢权重表示,缺乏推理过程中快速适应的能力。生物神经网络则通过快速突触更新形成临时关联记忆,称为赫布可塑性。本文对多种Transformer骨干网络(ViT-Small、DeiT-Small、Swin-Tiny)集成赫布快速权重(HFW)模块进行了实证研究。在元学习框架下,使用Prototypical Network在Omniglot数据集上评估五类一次和五类五次分类任务,共六种模型变体:ViT、DeiT、Swin、ViT-Hebbian、DeiT-Hebbian、Swin-Hebbian。提出一种单模块放置策略:仅在Swin-Tiny最终层级特征图上应用一个HFW模块,避免多层级引入导致的训练不稳。该设计在所有模型中取得最高测试准确率(1次学习96.2%;5次学习99.2%),较非赫布基线提升0.3个百分点。分析了Swin的移位窗口归纳偏置与片段级赫布绑定的交互关系,解释了为何在低数据条件下,维特和德伊特模型的逐块部署会失败,并将结果置于快速与慢速权重元学习文献背景中。
原文摘要 · Abstract (English)
Standard transformer architectures learn fixed slow-weight representations during training and lack mechanisms for rapid adaptation within an episode. In contrast, biological neural systems address this through fast synaptic updates that form transient associative memories during inference, a property known as Hebbian plasticity. In this paper, we conduct an empirical study of Hebbian Fast-Weight (HFW) modules integrated into multiple transformer backbones, including ViT-Small, DeiT-Small, and Swin-Tiny. We evaluate six model variants: ViT, DeiT, Swin, ViT-Hebbian, DeiT-Hebbian, and Swin-Hebbian on 5-way 1-shot and 5-way 5-shot classification tasks using the Omniglot benchmark under a Prototypical Network meta-learning framework. We propose a single module placement strategy for Swin-Tiny in which one HFW module is applied to the final stage feature map after all hierarchical stages have completed. This design avoids the training instability caused by placing separate Hebbian modules at each stage and achieves the highest test accuracy across all six models (96.2\% at 1-shot; 99.2\% at 5-shot), outperforming its non-Hebbian baseline by $+0.3$ percentage points at 1-shot. We analyze the interaction between Swin's shifted window inductive bias and episode-level Hebbian binding, discuss why per-block placement fails for ViT and DeiT variants in a low-data regime, and situate the results within the wider literature on fast and slow-weight meta-learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。