通过粒子优化学习注意力核函数，提升Transformer性能
A Coulomb Particle Model for Learning Kernel Attention in Transformers
- 用带排斥势的粒子模型优化随机特征分布
- 在多个任务上提升准确率、校准性和鲁棒性
- 保持线性注意力复杂度，适合高效部署
随机特征可对核机器进行可扩展近似，但其性能高度依赖特征分布的选择。我们提出一种基于粒子的方法，通过优化核-目标对齐并引入瑞斯/库仑排斥势来正则化粒子，从而得到具有多样性和任务自适应性的随机特征。该方法的哈密顿量可通过麦肯-弗拉索夫方程进行平均场描述。我们在线性化Transformer注意力中实现该方法：先在第一阶段学习正随机特征映射，再冻结核函数，用交叉熵训练剩余网络参数。在合成分类和句子级基准测试中，所学核化注意力在多种特征映射下均提升了准确率、校准性和鲁棒性，同时保持线性注意力的推理复杂度。
原文摘要 · Abstract (English)
Randomized features provide a scalable approximation to kernel machines, but their performance depends strongly on the choice of feature distribution. We propose a particle-based method that learns this distribution by optimizing kernel-target alignment while regularizing particles with a Riesz/Coulomb repulsive potential. The resulting Hamiltonian yields diverse, task-adaptive random features and admits a mean-field description through a McKean--Vlasov equation. We instantiate the method in linearized Transformer attention by learning positive random-feature maps in a first alignment phase, then freezing the kernel and training the remaining network parameters with cross-entropy. Experiments on synthetic classification and sentence-level benchmarks show that learned kernelized attention can improve accuracy, calibration, and robustness for several feature maps while preserving linear-attention inference complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。