arXiv:2605.16112cs.LGcs.AI2026-05

发现动态图Transformer在时间偏移下注意力分散,提出可迁移的改进方法。

Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix

论文配图:Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix
图 1 · 摘自论文原文
  • 用差分注意力抑制共现干扰,强化关键节点信号
  • 在9个基准上实现新最优,时间偏移越大提升越明显
  • 适合处理时间分布漂移的动态图预测任务

基于Transformer的连续时间动态图学习架构虽已成主流,但在时间分布偏移的数据集上性能仍受限。本文识别出注意力分散是动态图Transformer在时间偏移下的共性失效模式。通过控制消融实验,对比结构和时间上不同的历史邻居与随机邻居,发现预测依赖于一类携带更强预测信号的关键节点。然而现有Transformer即使输入中包含这些节点,也无法聚焦,因时间偏移削弱了注意力对比,导致注意力分布过度分散。据此提出简单且可迁移的解决方案:用差分注意力替代标准注意力,抑制共现注意力并放大个体标记信号。该方法在三个代表性动态图Transformer基线中均显著提升性能,增益集中在高偏移数据集上。注意力层级测量进一步验证机制,显示注意力熵降低、关键节点注意力质量提高。基于此,我们提出DiffDyG,结合差分注意力与标准输入编码。在9个基准和三种负采样策略下,其表现达到当前最优,尤其在最偏移的数据集上提升显著。

原文摘要 · Abstract (English)

Transformer-based architectures have become the dominant paradigm for Continuous-Time Dynamic Graph (CTDG) learning, yet their performance remains limited on temporally shifted datasets. In this work, we identify attention dispersion as a shared failure mode of dynamic graph Transformers under temporal distribution shift. Through controlled ablation contrasting structurally and temporally distinguished historical neighbors against random ones, we show that prediction depends on a class of critical nodes that carry consistently more predictive signal than arbitrary neighbors. However, existing Transformers fail to focus on these nodes even when they are present in the input, as temporal shift weakens attention contrast and produces overly dispersed attention distributions. This diagnosis suggests a simple and transferable fix: replace standard attention with differential attention, which suppresses common-mode attention and amplifies distinctive token-level signals. When added to three representative CTDG Transformer baselines, differential attention consistently improves performance, with gains concentrated on high-shift datasets. Attention-level measurements further confirm the mechanism, showing reduced attention entropy and increased attention mass on critical nodes. Building on these findings, we introduce DiffDyG, a reference implementation combining differential attention with standard input encodings. Across 9 benchmarks and three negative sampling protocols, DiffDyG achieves SOTA performance, with especially large gains on the most shifted datasets.

动态图注意力机制时间偏移Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。