提出线性复杂度的随机分批注意力,提升图Transformer内存效率与并行性。
How Particle-System Random Batch Methods Enhance Graph Transformer: Memory Efficiency and Parallel Computing Strategy
- 用随机分批策略替代传统注意力,实现线性时间复杂度。
- 在大规模图数据上验证了性能优势,内存占用显著降低。
- 理论支持收敛性,适合追求高效推理的图神经网络研究者。
注意力机制是Transformer模型的核心,通过全局信息增强嵌入向量的特征表达,其强大表征能力已被证实。然而,其二次时间复杂度限制了实际应用。尽管已有研究提出稀疏注意力形式以降低复杂度,但缺乏对其表征能力的理论分析。本文提出随机分批注意力(RBA),一种具有理论支撑的线性自注意力机制。RBA具备三大优势:(1) 时间复杂度为线性,且可在新维度上并行实现,显著节省内存;(2) 可无缝替换现有模型中的注意力模块,包括多种改进型注意力机制;(3) 其理论基础源于计算数学中的随机分批方法,具备收敛性证明。大规模图上的实验验证了上述优势。此外,该机制为未来注意力机制分析提供了新的理论工具。
原文摘要 · Abstract (English)
Attention mechanism is a significant part of Transformer models. It helps extract features from embedded vectors by adding global information and its expressivity has been proved to be powerful. Nevertheless, the quadratic complexity restricts its practicability. Although several researches have provided attention mechanism in sparse form, they are lack of theoretical analysis about the expressivity of their mechanism while reducing complexity. In this paper, we put forward Random Batch Attention (RBA), a linear self-attention mechanism, which has theoretical support of the ability to maintain its expressivity. Random Batch Attention has several significant strengths as follows: (1) Random Batch Attention has linear time complexity. Other than this, it can be implemented in parallel on a new dimension, which contributes to much memory saving. (2) Random Batch Attention mechanism can improve most of the existing models by replacing their attention mechanisms, even many previously improved attention mechanisms. (3) Random Batch Attention mechanism has theoretical explanation in convergence, as it comes from Random Batch Methods on computation mathematics. Experiments on large graphs have proved advantages mentioned above. Also, the theoretical modeling of self-attention mechanism is a new tool for future research on attention-mechanism analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。