用能量原理统一解释注意力机制，提出新型优化型注意力结构。
Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principle
- 基于能量函数构建统一框架，推导出多种注意力形式的生成逻辑。
- 将梯度下降、动量法等优化算法引入注意力设计，提升模型表达能力。
- 为注意力机制设计提供新思路，适合关注模型可解释性与架构创新的研究者。
基于注意力的Transformer在众多任务中表现出强大适应性，已成为现代大语言模型的核心。然而其内在机制仍有待深入探索。能量理论长期以来为神经计算提供了重要视角。本文以能量原理为切入点，提出一个包含局部能量 $E_i$、全局能量 $F$ 与优化算法三要素的统一能量框架。我们证明，未归一化的线性注意力、门控线性注意力及标准softmax注意力均可通过该框架中的不同配方诱导得出。在此基础上，我们提出能量驱动的注意力结构改进方案：受经典梯度下降（GD）启发，将原始注意力扩展至基于动量的GD、Nesterov加速梯度（NAG）及牛顿法，分别对应新型注意力结构。实验初步验证了该能量框架在设计注意力机制方面的潜力。
原文摘要 · Abstract (English)
Attention-based Transformers have demonstrated strong adaptability across a wide range of tasks and have become the backbone of modern Large Language Models (LLMs). However, their underlying mechanisms remain open for further exploration. The energy-based perspective has long provided a valuable principle for understanding neural computation. In this paper, we revisit the principle of energy as a lens to understand attention-based Transformer models. We present a unified energy-based framework which is composed of three key components: the local energy $E_i$, the global energy $F$, and the employed optimization algorithms. We show that different attention forms including unnormalized linear attention, gated linear attention and standard softmax attention can be induced by choosing their corresponding recipes within this framework. Building on this framework, we propose energy-based modifications of attention structures. Inspired by classical gradient descent (GD) algorithms, we extend the original attention formulation based on standard GD to the momentum-based GD, Nesterov Accelerated Gradient (NAG), and Newton's method, each inducing a corresponding new attention structure. Our experiments provide preliminary support for the potential of the energy-based framework for designing attention mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。