对比稀疏与稠密注意力，揭示图Transformer的适用场景
Scaling Graph Transformers: A Comparative Study of Sparse and Dense Attention
- 比较图Transformer中稠密与稀疏注意力的机制差异
- 发现稀疏注意力在长程依赖建模上更高效
- 适合关注图神经网络注意力设计的研究者
图已成为机器学习中表示关系与结构数据的核心范式，广泛应用于多个领域。传统图神经网络受限于局部结构，难以捕捉节点间的长程依赖。图Transformer通过注意力机制实现全局信息交换，克服了这一局限。然而，图Transformer中存在两种注意力形式：稠密注意力与稀疏注意力。本文系统比较这两种机制，分析其权衡关系，并明确各自的适用场景。同时，论文梳理了当前图Transformer注意力设计中的关键挑战与待解决问题。
原文摘要 · Abstract (English)
Graphs have become a central representation in machine learning for capturing relational and structured data across various domains. Traditional graph neural networks often struggle to capture long-range dependencies between nodes due to their local structure. Graph transformers overcome this by using attention mechanisms that allow nodes to exchange information globally. However, there are two types of attention in graph transformers: dense and sparse. In this paper, we compare these two attention mechanisms, analyze their trade-offs, and highlight when to use each. We also outline current challenges and problems in designing attention for graph transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。