提升线性图注意力的表达能力与聚焦精度,兼顾效率与性能。
GraphTARIF: Linear Graph Transformer with Augmented Rank and Improved Focus
- 通过附加门控局部图网络增强注意力矩阵秩
- 引入可学习的对数幂函数降低注意力熵,提升聚焦度
- 在同质与异质图上均实现高效高精度,适合大规模图分析
线性注意力机制已成为图变压器中全自注意力的高效替代方案,具备线性时间复杂度。然而,现有线性注意力模型常因低秩投影结构和过于均匀的注意力分布导致表达能力下降。我们从理论上证明,这些特性会降低节点表示的类别可分性,限制分类能力。为此,我们提出一种新型混合框架,同时增强注意力的秩与聚焦度。具体而言,通过在值矩阵上附加一个门控局部图网络分支,提升注意力映射的秩;此外,为缓解线性注意力固有的过度平滑问题,引入可学习的对数幂函数到注意力分数中,以降低熵并增强聚焦。理论分析表明,该函数能减少注意力分布的熵,提升嵌入表示的可分性。在同质与异质图基准数据集上的大量实验表明,本方法在保持线性注意力可扩展性的前提下,实现了具有竞争力的性能。
原文摘要 · Abstract (English)
Linear attention mechanisms have emerged as efficient alternatives to full self-attention in Graph Transformers, offering linear time complexity. However, existing linear attention models often suffer from a significant drop in expressiveness due to low-rank projection structures and overly uniform attention distributions. We theoretically prove that these properties reduce the class separability of node representations, limiting the model's classification ability. To address this, we propose a novel hybrid framework that enhances both the rank and focus of attention. Specifically, we enhance linear attention by attaching a gated local graph network branch to the value matrix, thereby increasing the rank of the resulting attention map. Furthermore, to alleviate the excessive smoothing effect inherent in linear attention, we introduce a learnable log-power function into the attention scores to reduce entropy and sharpen focus. We theoretically show that this function decreases entropy in the attention distribution, enhancing the separability of learned embeddings. Extensive experiments on both homophilic and heterophilic graph benchmarks demonstrate that our method achieves competitive performance while preserving the scalability of linear attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。