提出一种统一方法，让注意力机制既可解释又高效。
Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few
- 通过收缩少数代表令牌压缩输入，实现线性计算复杂度。
- 在视觉任务中表现相当，但比黑箱注意力更高效可解释。
- 适合关注模型透明性与推理效率的研究者。
注意力机制在多个领域取得了显著的实证成功，但其底层优化目标仍不明确。此外，自注意力的二次复杂度日益成为瓶颈。尽管可解释性与效率是相互促进的目标，但以往工作通常分别研究。本文提出一个统一优化目标，通过算法展开推导出内在可解释且高效的注意力机制。具体而言，构建了该目标的梯度步骤，对应我们提出的 extit{Contract-and-Broadcast Self-Attention}（CBSA）前向操作：通过压缩少数代表性令牌，将输入令牌聚合到低维结构。该机制在固定代表性令牌数量时可实现线性扩展，并能覆盖不同代表性集合下的多种注意力实例。大量实验表明，其在视觉任务上性能相当，但优于黑箱注意力机制，在效率和可解释性方面优势明显。本工作为可解释性与效率的融合提供了新思路，并揭示了注意力机制的统一公式。
原文摘要 · Abstract (English)
Attention mechanisms have achieved significant empirical success in multiple fields, but their underlying optimization objectives remain unclear yet. Moreover, the quadratic complexity of self-attention has become increasingly prohibitive. Although interpretability and efficiency are two mutually reinforcing pursuits, prior work typically investigates them separately. In this paper, we propose a unified optimization objective that derives inherently interpretable and efficient attention mechanisms through algorithm unrolling. Precisely, we construct a gradient step of the proposed objective with a set of forward-pass operations of our \emph{Contract-and-Broadcast Self-Attention} (CBSA), which compresses input tokens towards low-dimensional structures by contracting a few representatives of them. This novel mechanism can not only scale linearly by fixing the number of representatives, but also covers the instantiations of varied attention mechanisms when using different sets of representatives. We conduct extensive experiments to demonstrate comparable performance and superior advantages over black-box attention mechanisms on visual tasks. Our work sheds light on the integration of interpretability and efficiency, as well as the unified formula of attention mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。