arXiv:2605.22471cs.LG2026-05被引 2

图转换器的性能取决于图分词方式,不同分词有根本性差异。

Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers

论文配图:Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
图 1 · 摘自论文原文
  • 对比谱、随机游走和邻接三种图分词方法
  • 同一任务在不同分词下所需模型深度相差显著
  • 随机游走分词会丢失信息,谱分词对局部任务不友好

Transformer已成为图学习的核心架构,但其应用需先选择图分词方式——即图到标记的映射,决定输入中暴露的结构信息。本文揭示该选择是Transformer表达能力的根本因素。我们分析三种作为现有图分词基础的构建块:谱分词、随机游走分词和邻接分词。证明不同分词诱导出不同的深度区间:相同图计算在一种分词下可用浅层Transformer实现,而在另一种下则需大幅增加深度。例如,我们证明随机游走分词对任意游走长度均具有损失性,一般无法从中恢复原图;而谱分词虽无损,却对局部任务条件不佳。进一步表明,尽管随机游走与谱分词均源自邻接信息,但有限深度的Transformer通常无法在分词族间转换。我们建立下界与不可能性结果,显示不良分词可能阻碍高效恢复更优结构表示。最后,通过合成与真实世界任务的受控实验验证理论预测,显示不同任务偏好不同结构视图,结合互补分词可使Transformer利用各表示的独特信号。

原文摘要 · Abstract (English)

Transformers have become a central architecture for graph learning, but their application to graphs requires first choosing a tokenization: a graph-to-token map that determines which structural information is exposed at the input. In this work, we show that this choice is a fundamental component of transformer expressivity. We examine three tokenizations that serve as building blocks for many existing graph tokenizations: spectral, random-walk, and adjacency tokenizations. We prove that different tokenizations induce distinct depth regimes: the same graph computation may be realizable by a shallow transformer under one tokenization, while requiring substantially larger depth under another. For example, we prove that random-walk tokenization is lossy for any walk length, making it impossible in general to recover the graph from it, and that while spectral tokenization is lossless, it is ill-conditioned for local tasks. We further show that although both random-walk and spectral tokenizations are derived from adjacency information, it is impossible for a limited-depth transformer to convert between tokenization families in general. In particular, we establish lower bounds and impossibility results showing that unfavorable tokenizations may preclude the efficient recovery of more suitable structural representations. Finally, we complement our theory with controlled experiments on synthetic and real-world tasks, validating the predicted separations and showing that different tasks favor different structural views, and combining complementary tokenizations allows the transformer to leverage distinct signals from each representation.

图神经网络注意力机制结构表示分词策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。