多头注意力通过协同计算图提升信息传播效率，超越单纯并行。
Beyond Parallelism: Synergistic Computational Graph Effects in Multi-Head Attention
- 将多头注意力视为共享汇点的协同计算图系统
- 在头间多样性条件下，混合时间更快、信息保真度更高
- 实验证明同参数量下多头优于单头，适合研究模型机制者
多头注意力是驱动大语言模型成功的Transformer架构核心。然而，多头相对于单头的理论优势，除了并行处理外仍缺乏深入探索。本文将多头注意力重新建模为一组可能产生协同效应的计算图系统，其中每个头是一个以共同汇点状态为终点的前馈有向无环图（DAG）。我们提供了关于混合时间与最小最大保真度的直观解释和初步理论分析。结果表明，在特定头间多样性条件下，多头注意力可协同增强信息传播，实现更短的混合时间和更高的最小最大保真度。最后，我们在序列操作任务上训练了参数总量相同的单头与多头Transformer，并实证验证了上述预测效果。代码已公开于https://github.com/haitzsaezdeocariz/beyondparallelism。
原文摘要 · Abstract (English)
Multi-head attention powers Transformer networks, the primary deep learning architecture behind the success of large language models (LLMs). Yet, the theoretical advantages of multi-head versus single-head attention, beyond mere parallel processing, remain underexplored. In this paper, we reframe multi-head attention as a system of potentially synergistic computational graphs, where each head functions as a feedforward directed acyclic graph (DAG) with a common sink state. We provide intuition and preliminary theoretical analysis of mixing time and minimax fidelity in this framework. Our results show that multi-head attention can synergistically enhance information propagation, yielding faster mixing times and minimax fidelity amplification under specific head-diversity conditions. Finally, we train single-head and multi-head Transformers, each with the same total number of parameters, on sequence manipulation tasks and empirically verify the predicted effects. The code is available at https://github.com/haitzsaezdeocariz/beyondparallelism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。