用可路由的低秩专家块替换Transformer的前馈网络，缓解持续学习中的灾难性遗忘。
Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts
- 通过相似度路由选择特定词元对应的低秩专家模块进行计算
- 在字符级语言建模中保持相近困惑度（4.57 vs 4.32），但持续适应能力提升2.6~2.8倍
- 适合需要长期更新、避免权重干扰的持续学习场景
Transformer 在语言建模中表现优异，但其位置前馈网络（FFN）为稠密全局共享结构，导致计算资源浪费且易受持续学习中权重更新干扰。本文提出吸引子块网络（APN），作为Transformer FFN的即插即用替代方案。APN由一组块专家构成，通过相似度路由机制，根据词元表示匹配预训练原型，选出最相关的少量块。每个选中块生成基于紧凑编码的低秩残差更新。该架构实现上下文自适应的非线性变换，同时保持标准Transformer接口。本文形式化了APN，分析其作为分段低秩残差函数类的表达能力，并推导出简洁的干扰与稳定性论证，表明其天然适合持续学习。在字符级语言建模实验中，APN取得4.57的困惑度（对比基线4.32），在域迁移任务中，原域保留能力提升2.6倍（11.1 vs 29.4 PPL），新域适应能力提升2.8倍（6.4 vs 17.8 PPL）。
原文摘要 · Abstract (English)
Transformers achieve strong language modeling accuracy, yet their position-wise feed-forward networks (FFNs) are dense, globally shared, and typically updated end to end. These properties create two practical tensions. First, dense FFNs spend the same compute on every token regardless of context, and they allocate capacity uniformly even when language exhibits highly clustered context structure. Second, continual learning, in the sense of updating the model while serving a data stream, often produces interference because a small update touches broadly shared weights. We propose Attractor Patch Networks (APN), a plug-compatible replacement for the Transformer FFN. APN is a bank of patch experts. A similarity router selects a small top-k set of patches for each token by matching the token representation to learned prototypes. Each selected patch emits a low-rank residual update conditioned on a compact code. The architecture yields conditional, context-specialized nonlinear transformations while preserving the standard Transformer interface. This paper focuses on APN as an architectural primitive. We formalize APN, analyze its expressivity as a piecewise low-rank residual function class, and derive simple interference and stability arguments that make APN naturally compatible with continual learning. In experiments on character-level language modeling, APN achieves competitive perplexity (4.57 vs 4.32 PPL) while enabling dramatically better continual adaptation: when adapting to a shifted domain, APN achieves 2.6 times better retention (11.1 vs 29.4 PPL on the original domain) and 2.8 times better adaptation (6.4 vs 17.8 PPL on the new domain) compared to global fine-tuning of a dense FFN baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。