提出新型波动注意力机制，缓解深度Transformer过平滑问题。
Wavy Transformer
- 将注意力层动态类比为图上扩散过程，揭示过平滑本质
- 引入二阶波动机制，显著提升模型表征能力
- 无需额外调参，适配各类NLP/CV任务
Transformers在自然语言处理和计算机视觉领域取得了显著成功。然而，深层Transformer模型常面临过平滑问题，即随着注意力层堆叠，标记表示趋于相似。本文建立堆叠注意力层引发的隐藏状态动态与完全图上的图神经网络扩散之间的等价关系。从这一视角看，过平滑是底层扩散动力学耗散性的结果。受此物理启发，我们提出Wavy Transformer，其核心是基于二阶波动动力学的新颖注意力层。同时设计了保持链式法则下物理态-速度关系的前馈网络与归一化层，从而扩展Transformer架构。我们在多种NLP和CV任务的Transformer模型上验证了所提方法，结果一致表明，Wavy Transformer在几乎不增加参数量且无需额外超参数调优的情况下，显著提升性能。
原文摘要 · Abstract (English)
Transformers have achieved remarkable success across natural language processing (NLP) and computer vision (CV). However, deep transformer models often suffer from an over-smoothing issue, in which token representations converge to similar values as they pass through successive transformer blocks. In this paper, we establish an equivalence between the hidden-state dynamics induced by stacked attention layers and graph neural diffusion on a complete graph. From this perspective, over-smoothing can be interpreted as a consequence of the dissipative nature of the underlying diffusion dynamics. Motivated by this physical interpretation, we propose Wavy Transformer, which consists of a novel attention layer based on second-order wavy dynamics. We also introduce a feed-forward network and a normalization layer designed to preserve the physical state-velocity relationship under the chain rule, thereby extending the transformer architecture. We further validate our proposed techniques on various transformer models for NLP and CV tasks. The results consistently demonstrate that Wavy Transformer improves performance with minimal additional parameters and no extra hyperparameter tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。