左优先注意力让Transformer表达能力变弱,影响模型设计选择。
Unique Hard Attention: A Tale of Two Sides
- 区分左/右优先硬注意力,揭示其对模型表达力的决定性影响。
- 仅用左优先注意力时,模型等价于更弱的LTL片段,无法表达某些逻辑模式。
- 左优先模型与软注意力等价,可能更贴近真实模型行为,适合理论研究。
理解Transformer的表达能力近年来备受关注,因其能揭示其能力与局限。现有研究多聚焦于唯一硬注意力机制,即注意力选择得分最高的单一位置;当多个位置得分并列最大时,通常取最左或最右的位置。本文强调这一看似微小的差异的重要性。近期研究表明,采用左右两种硬注意力的有限精度Transformer等价于线性时序逻辑(LTL)。我们发现,若仅使用左优先硬注意力,则其不再等价于完整LTL,而是对应一个严格更弱的LTL子集。此外,我们证明左优先注意力模型等价于软注意力,表明其可能比右优先模型更准确地逼近实际应用中的Transformer。这些发现细化了Transformer表达力的理论图景,并凸显了注意力方向性在建模中的关键作用。
原文摘要 · Abstract (English)
Understanding the expressive power of transformers has recently attracted attention, as it offers insights into their abilities and limitations. Many studies analyze unique hard attention transformers, where attention selects a single position that maximizes the attention scores. When multiple positions achieve the maximum score, either the rightmost or the leftmost of those is chosen. In this paper, we highlight the importance of this seeming triviality. Recently, finite-precision transformers with both leftmost- and rightmost-hard attention were shown to be equivalent to Linear Temporal Logic (LTL). We show that this no longer holds with only leftmost-hard attention -- in that case, they correspond to a \emph{strictly weaker} fragment of LTL. Furthermore, we show that models with leftmost-hard attention are equivalent to \emph{soft} attention, suggesting they may better approximate real-world transformers than right-attention models. These findings refine the landscape of transformer expressivity and underscore the role of attention directionality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。