提出图量化分词器GQT，让图Transformer更高效通用。
Learning Graph Quantized Tokenizers
- 用自监督学习分离训练分词器与Transformer，提升泛化性。
- 采用残差向量量化生成分层离散令牌，内存大幅降低。
- 搭配令牌调制，在22个基准中20个达顶尖性能，适合图学习任务。
Transformers是基础模型的核心架构，领域专用分词器使其能适配多种场景。图变压器（GT）近年来成为几何深度学习的领先模型，在各类图学习任务中表现优于图神经网络（GNN）。然而，图分词器的发展滞后于其他模态。为此，我们提出GQT（Graph Quantized Tokenizer），通过多任务图自监督学习将分词器训练与Transformer训练解耦，获得鲁棒且通用的图令牌。此外，GQT利用残差向量量化（RVQ）学习分层离散令牌，显著降低内存需求并提升泛化能力。结合令牌调制，该方法在22个基准中的20个上实现最优性能，涵盖大规模同质与异质数据集。
原文摘要 · Abstract (English)
Transformers serve as the backbone architectures of Foundational Models, where domain-specific tokenizers allow them to adapt to various domains. Graph Transformers (GTs) have recently emerged as leading models in geometric deep learning, outperforming Graph Neural Networks (GNNs) in various graph learning tasks. However, the development of tokenizers for graphs has lagged behind other modalities. To address this, we introduce GQT (\textbf{G}raph \textbf{Q}uantized \textbf{T}okenizer), which decouples tokenizer training from Transformer training by leveraging multi-task graph self-supervised learning, yielding robust and generalizable graph tokens. Furthermore, the GQT utilizes Residual Vector Quantization (RVQ) to learn hierarchical discrete tokens, resulting in significantly reduced memory requirements and improved generalization capabilities. By combining the GQT with token modulation, a Transformer encoder achieves state-of-the-art performance on 20 out of 22 benchmarks, including large-scale homophilic and heterophilic datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。