arXiv:2412.21046cs.LG2024-12被引 1

发现递归图模型因截断反向传播导致性能下降,影响长期依赖学习。

Mind the truncation gap: challenges of learning on dynamic graphs with recurrent architectures

  • 提出截断反向传播对动态图模型的限制机制
  • 实证显示截断训练比完整反向传播性能更低
  • 适合研究动态图建模与序列学习的学者参考

具有演化交互的系统在社交、金融和生物领域普遍存在,可有效建模为连续时间动态图(CTDGs)。为应对这类图数据的规模与复杂性,机器学习方法至关重要。然而,传统静态图方法无法自然处理事件时间信息。新型图循环神经网络(GRNN)虽具备时间感知能力,但面临反向传播通过时间(BPTT)截断问题,其影响尚未被充分研究。本文揭示,短截断会限制模型学习单跳以外的依赖关系,导致性能下降。在新设计的合成任务与真实数据集上的实验表明,完整反向传播(F-BPTT)与常用截断反向传播(T-BPTT)之间存在显著性能差距,称之为“截断差距”。我们强调理解并解决该问题对日益重要的CTDG建模意义重大,并讨论未来研究方向。

原文摘要 · Abstract (English)

Systems characterized by evolving interactions, prevalent in social, financial, and biological domains, are effectively modeled as continuous-time dynamic graphs (CTDGs). To manage the scale and complexity of these graph datasets, machine learning (ML) approaches have become essential. However, CTDGs pose challenges for ML because traditional static graph methods do not naturally account for event timings. Newer approaches, such as graph recurrent neural networks (GRNNs), are inherently time-aware and offer advantages over static methods for CTDGs. However, GRNNs face another issue: the short truncation of backpropagation-through-time (BPTT), whose impact has not been properly examined until now. In this work, we demonstrate that this truncation can limit the learning of dependencies beyond a single hop, resulting in reduced performance. Through experiments on a novel synthetic task and real-world datasets, we reveal a performance gap between full backpropagation-through-time (F-BPTT) and the truncated backpropagation-through-time (T-BPTT) commonly used to train GRNN models. We term this gap the "truncation gap" and argue that understanding and addressing it is essential as the importance of CTDGs grows, discussing potential future directions for research in this area.

动态图序列建模深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。