arXiv:2505.24054cs.LG2025-05被引 3

通过生物启发的门控机制,提升Transformer对噪声输入的鲁棒性

Differential Gated Self-Attention

  • 每头分兴奋与抑制分支,用门控融合实现上下文感知的注意力增强
  • 在视觉与语言任务上均显著优于标准Transformer和差分Transformer
  • 适合需要高鲁棒性的实际场景,如含噪图像识别或低质量文本处理

Transformer在众多任务中表现优异,但对噪声输入仍敏感,因标准自注意力对所有查询-键交互一视同仁。受生物神经回路侧抑制启发,并基于差分Transformer利用双并行softmax相减实现噪声抑制的思路,我们提出多头差分门控自注意力(M-DGSA),通过学习每头的输入相关门控,动态抑制注意力噪声。每个头分为兴奋与抑制分支,其双softmax映射通过由标记嵌入预测的sigmoid门控融合,实现上下文感知的对比度增强。M-DGSA可无缝集成至现有Transformer结构,计算开销极小。我们在视觉与语言基准上评估,结果表明其在鲁棒性上持续优于标准Transformer、Vision Transformer及差分Transformer基线。主要贡献包括:(i) 基于侧抑制的新型输入依赖门控机制;(ii) 生物对比增强与自注意力理论的原理性结合;(iii) 全面实验验证了其抗噪能力与跨领域适用性。

原文摘要 · Abstract (English)

Transformers excel across a large variety of tasks but remain susceptible to corrupted inputs, since standard self-attention treats all query-key interactions uniformly. Inspired by lateral inhibition in biological neural circuits and building on the recent use by the Differential Transformer's use of two parallel softmax subtraction for noise cancellation, we propose Multihead Differential Gated Self-Attention (M-DGSA) that learns per-head input-dependent gating to dynamically suppress attention noise. Each head splits into excitatory and inhibitory branches whose dual softmax maps are fused by a sigmoid gate predicted from the token embedding, yielding a context-aware contrast enhancement. M-DGSA integrates seamlessly into existing Transformer stacks with minimal computational overhead. We evaluate on both vision and language benchmarks, demonstrating consistent robustness gains over vanilla Transformer, Vision Transformer, and Differential Transformer baselines. Our contributions are (i) a novel input-dependent gating mechanism for self-attention grounded in lateral inhibition, (ii) a principled synthesis of biological contrast-enhancement and self-attention theory, and (iii) comprehensive experiments demonstrating noise resilience and cross-domain applicability.

自注意力抗噪生物启发Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。