arXiv:2604.17411cs.CLcs.AI2026-04

提出双粒度文本编码器,让文本理解更懂图结构。

DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs

论文配图:DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs
图 1 · 摘自论文原文
  • 分两步编码:先词粒度,再节点粒度,融合图结构引导注意力。
  • 在多个基准数据集上达到当前最优效果,提升显著。
  • 适合处理带文本的图数据,如文档分类、信息抽取任务。

文本属性图将节点文本的语义信息与拓扑结构相结合,在文档分类、信息抽取等应用中具有重要价值。现有方法通常使用语言模型(LM)编码文本内容,再通过图神经网络(GNN)处理结构信息。然而,在基于LM的文本编码阶段,多数方法仅在词元粒度上进行语义交互,并忽视了不同节点间文本的结构依赖关系。本文提出DuConTE,一种带有拓扑约束注意力的双粒度文本编码器。该模型采用两级预训练语言模型级联架构,先在词元粒度编码语义,再在节点粒度建模。在每个语言模型的自注意力计算中,动态调整注意力掩码矩阵,根据节点连通性引导模型学习受图结构启发的语义关联。此外,在组合节点表示时,分别评估词元在中心节点上下文和邻域上下文中的重要性,从而捕捉更相关的上下文语义信息。在多个基准数据集上的大量实验表明,DuConTE在多数数据集上达到领先性能。

原文摘要 · Abstract (English)

Text-attributed graphs integrate semantic information of node texts with topological structure, offering significant value in various applications such as document classification and information extraction. Existing approaches typically encode textual content using language models (LMs), followed by graph neural networks (GNNs) to process structural information. However, during the LM-based text encoding phase, most methods not only perform semantic interaction solely at the word-token granularity, but also neglect the structural dependencies among texts from different nodes. In this work, we propose DuConTE, a dual-granularity text encoder with topology-constrained attention. The model employs a cascaded architecture of two pretrained LMs, encoding semantics first at the word-token granularity and then at the node granularity. During the self-attention computation in each LM, we dynamically adjust the attention mask matrix based on node connectivity, guiding the model to learn semantic correlations informed by the graph structure. Furthermore, when composing node representations from word-token embeddings, we separately evaluate the importance of tokens under the center-node context and the neighborhood context, enabling the capture of more contextually relevant semantic information. Extensive experiments on multiple benchmark datasets demonstrate that DuConTE achieves state-of-the-art performance on the majority of them.

图神经网络文本编码双粒度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。