通过随机遮蔽注意力分布，提升Transformer模型的泛化与鲁棒性。
AttentionDrop: A Novel Regularization Method for Transformer Models
- 在注意力权重上施加三种随机正则化，增强上下文利用多样性。
- 相比标准Dropout等方法，准确率、校准度和抗攻击能力全面提升。
- 适合数据少或噪声大的场景，尤其适用于高风险应用。
基于Transformer的模型在自然语言处理、计算机视觉和语音处理等多个领域表现优异，但其强大容量常导致过拟合，尤其是在训练数据有限或含噪时。本文提出统一的随机正则化方法族——AttentionDrop，包含三种变体：硬注意力掩码（Hard Attention Masking）对每个查询的前k个注意力logits随机置零，以鼓励多样化的上下文利用；模糊注意力平滑（Blurred Attention Smoothing）对注意力logits施加动态高斯卷积，使过于尖锐的分布趋于平滑；一致性正则化注意力丢弃（Consistency-Regularized AttentionDrop）通过基于KL散度的一致性损失，强制在多次独立AttentionDrop扰动下输出保持稳定。实验结果表明，AttentionDrop在准确率、校准性和对抗鲁棒性方面均显著优于标准Dropout、DropConnect和R-Drop基线。
原文摘要 · Abstract (English)
Transformer-based architectures achieve state-of-the-art performance across a wide range of tasks in natural language processing, computer vision, and speech processing. However, their immense capacity often leads to overfitting, especially when training data is limited or noisy. In this research, a unified family of stochastic regularization techniques has been proposed, i.e. AttentionDrop with its three different variants, which operate directly on the self-attention distributions. Hard Attention Masking randomly zeroes out top-k attention logits per query to encourage diverse context utilization, Blurred Attention Smoothing applies a dynamic Gaussian convolution over attention logits to diffuse overly peaked distributions, and Consistency-Regularized AttentionDrop enforces output stability under multiple independent AttentionDrop perturbations via a KL-based consistency loss. Results achieved in the study demonstrate that AttentionDrop consistently improves accuracy, calibration, and adversarial robustness over standard Dropout, DropConnect, and R-Drop baselines
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。