让Transformer的注意力头各用不同范围的上下文，提升模型效率与可解释性。
Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention
- 按头分组并分配不同长度的局部上下文窗口，实现结构化注意力分配。
- 4096标记实验中，部分变体验证损失低于全注意力，证明有效降低冗余。
- 适合追求模型轻量化与注意力机制可解释性的研究者使用。
标准多头注意力（MHA）为每个头提供相同的完整因果上下文跨度，尽管各头可能承担不同上下文角色。有些头依赖近邻词汇或句法信息，而另一些则需长距离关系如实体交互、话语连接或状态变化。本文提出非对称注意力头（AAH），一种头级上下文分配框架，将上下文长度作为每个头或每组的显式分配变量。AAH基于特征统计对头进行分组，构建层次化结构，并分配因果局部窗口，同时保持标准扁平式MHA输出接口。在4096标记种子0实验中，多种AAH风格的局部分配变体实现比纯全注意力更低的验证损失。短预算消融实验表明，稳定的局部分配与头-窗口分配结构至关重要，而固定/局部控制策略也可与自适应层级方法竞争。我们把AAH解释为一种结构化的头级上下文分配机制，用于质量评估与分析，报告了注意力覆盖比（ACR）作为选择窗口路由的诊断指标。
原文摘要 · Abstract (English)
Standard multi-head attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby lexical or syntactic context, while others may depend on longer-range relations such as entity interactions, discourse links, or state changes. We present Asymmetric Attention Heads (AAH), a head-wise context- allocation framework that treats context length as an explicit per-head or per-group allocation variable. AAH groups heads using feature-derived statistics, organizes these groups hierarchically, and assigns causal local windows while preserving the standard flat MHA output interface. In 4096- token seed-0 experiments, several AAH-style local-allocation variants achieve lower validation loss than pure full attention. Short-budget ablations show that stable local allocation and head-window assignment structure matter, while fixed/local controls can be competitive with adaptive hierarchy. We interpret AAH as a structured head-wise context-allocation mechanism for quality and analysis, with Attention Coverage Ratio (ACR) reported as a selected-window routing diagnostic
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。