arXiv:2412.11293cs.LGcs.AI2024-12被引 5

对比Mamba与Transformer在动态图嵌入中的表现,发现Mamba更高效且在高变化数据上更优。

A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformers

  • 用Mamba替代Transformer,实现线性复杂度的动态图建模。
  • 在链接预测任务中,Mamba模型性能相当或更好,尤其在高时序变化数据上领先。
  • 适合需要处理长序列、大规模动态网络的研究者和应用者。

动态图嵌入已成为建模跨多领域时变网络的重要技术。尽管基于Transformer的模型在捕捉时间图数据中的长程依赖方面表现出色,但其二次计算复杂度带来可扩展性挑战。本研究对基于Transformer和新兴的线性复杂度状态空间模型Mamba的动态图嵌入方法进行了比较分析。我们提出了三种新模型:TransformerG2G增强图卷积网络,DG-Mamba,以及结合图同构网络边卷积的 GDG-Mamba。在多个基准数据集上的实验表明,基于Mamba的模型在链接预测任务中达到相当或更优性能,同时在长序列上显著提升计算效率。值得注意的是,DG-Mamba在高时序变异性数据集(如UCI、Bitcoin、Reality Mining)上持续优于Transformer模型,且在较稳定图(如SBM)上保持竞争力。通过分析注意力权重与状态矩阵,揭示了模型对复杂时间模式的捕捉能力。通过有效结合状态空间模型与图神经网络,本工作解决了先前方法的关键局限,推动了高效时序图表示学习的发展,为社交网络分析、金融建模及生物系统动力学等场景提供新可能。

原文摘要 · Abstract (English)

Dynamic graph embedding has emerged as an important technique for modeling complex time-evolving networks across diverse domains. While transformer-based models have shown promise in capturing long-range dependencies in temporal graph data, they face scalability challenges due to quadratic computational complexity. This study presents a comparative analysis of dynamic graph embedding approaches using transformers and the recently proposed Mamba architecture, a state-space model with linear complexity. We introduce three novel models: TransformerG2G augment with graph convolutional networks, \mathcal{DG}-Mamba, and \mathcal{GDG}-Mamba with graph isomorphism network edge convolutions. Our experiments on multiple benchmark datasets demonstrate that Mamba-based models achieve comparable or superior performance to transformer-based approaches in link prediction tasks while offering significant computational efficiency gains on longer sequences. Notably, \mathcal{DG}-Mamba variants consistently outperform transformer-based models on datasets with high temporal variability, such as UCI, Bitcoin, and Reality Mining, while maintaining competitive performance on more stable graphs like SBM. We provide insights into the learned temporal dependencies through analysis of attention weights and state matrices, revealing the models' ability to capture complex temporal patterns. By effectively combining state-space models with graph neural networks, our work addresses key limitations of previous approaches and contributes to the growing body of research on efficient temporal graph representation learning. These findings offer promising directions for scaling dynamic graph embedding to larger, more complex real-world networks, potentially enabling new applications in areas such as social network analysis, financial modeling, and biological system dynamics.

动态图Mamba图神经网络效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。