arXiv:2511.21416cs.CLcs.LG2025-11

将图结构注入Transformer,实现文本与拓扑的高效融合。

Odin: Oriented Dual-module Integration for Text-rich Network Representation Learning

  • 在特定层注入图结构,避免过平滑和多跳扩散依赖。
  • 在多个文本图基准上达到顶尖准确率,轻量版效率显著提升。
  • 适合需要融合文本与图结构的复杂网络分析任务。

文本属性图需同时具备强大的文本理解与结构化推理能力。现有方法或依赖受过平滑和跳数依赖限制的GNN,或使用忽略图拓扑的Transformer,将节点视为孤立序列。我们提出Odin(定向双模块集成),通过定向双模块机制在选定深度将图结构注入Transformer。不同于消息传递的GNN,Odin不依赖多跳扩散;而是将多跳结构整合于特定Transformer层,生成与语义层次对齐的低、中、高层结构抽象。因聚合基于全局[CLS]表示,Odin从根本上避免过平滑,并解耦结构抽象与邻域大小或图拓扑。我们进一步证明Odin的表达能力严格包含纯Transformer和GNN。为提升大规模或低资源场景效率,提出轻量版Light Odin,保持相同的层对齐结构抽象,实现更快训练与推理。多个文本图基准实验表明,Odin达到最先进准确率,Light Odin则以显著降低的计算开销实现竞争力表现。Odin与Light Odin共同构成一个统一、无跳数依赖的结构-文本融合框架。源码已发布于https://github.com/hongkaifeng/Odin。

原文摘要 · Abstract (English)

Text-attributed graphs require models to effectively combine strong textual understanding with structurally informed reasoning. Existing approaches either rely on GNNs--limited by over-smoothing and hop-dependent diffusion--or employ Transformers that overlook graph topology and treat nodes as isolated sequences. We propose Odin (Oriented Dual-module INtegration), a new architecture that injects graph structure into Transformers at selected depths through an oriented dual-module mechanism. Unlike message-passing GNNs, Odin does not rely on multi-hop diffusion; instead, multi-hop structures are integrated at specific Transformer layers, yielding low-, mid-, and high-level structural abstraction aligned with the model's semantic hierarchy. Because aggregation operates on the global [CLS] representation, Odin fundamentally avoids over-smoothing and decouples structural abstraction from neighborhood size or graph topology. We further establish that Odin's expressive power strictly contains that of both pure Transformers and GNNs. To make the design efficient in large-scale or low-resource settings, we introduce Light Odin, a lightweight variant that preserves the same layer-aligned structural abstraction for faster training and inference. Experiments on multiple text-rich graph benchmarks show that Odin achieves state-of-the-art accuracy, while Light Odin delivers competitive performance with significantly reduced computational cost. Together, Odin and Light Odin form a unified, hop-free framework for principled structure-text integration. The source code of this model has been released at https://github.com/hongkaifeng/Odin.

图神经网络文本生成Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。