arXiv:2605.10503cs.AI2026-05

让大模型自己看清图结构,无需训练就能提升图推理能力

SLASH the Sink: Sharpening Structural Attention Inside LLMs

论文配图:SLASH the Sink: Sharpening Structural Attention Inside LLMs
图 1 · 摘自论文原文
  • 发现大模型内部隐含图结构信息,通过注意力模式呈现锯齿状特征
  • 提出SLASH方法,无需训练即可增强模型对图拓扑的感知能力
  • 适合需要高效图推理的场景,如分子性质预测和图任务

大型语言模型(LLMs)在语义理解上表现卓越,但在以序列形式处理图拓扑时常缺乏结构理解。现有方案依赖外部图适配器或微调,成本高且泛化性差。本文研究发现:LLMs会自发重建图结构,其注意力图中存在与“令牌级邻接矩阵”结构对齐的“锯齿状”模式。然而,这种内在结构理解被注意力汇聚(attention sink)稀释。我们从理论上将此稀释归因于表示瓶颈,源于语言任务所需的各向异性偏差抑制了图推理所需的局部聚合。为此,提出无需训练的解决方案SLASH(StructuraL Attention SHarpening),通过即插即用的注意力重分配强化内部结构感知。在纯图任务与分子预测任务上的实验表明,SLASH在多种LLMs上均带来显著且一致的性能提升。

原文摘要 · Abstract (English)

Large Language Models (LLMs) show remarkable semantic understanding but often struggle with structural understanding when processing graph topologies in a serialized format. Existing solutions rely on training external graph-based adapters or fine-tuning, which incur high costs and lost generalizability. In this work, we investigate the internal mechanisms of LLMs and present a critical finding: LLMs spontaneously reconstruct the graph's topology internally, evidenced by a distinct "sawtooth" pattern in their attention maps that structurally aligns with the "token-level adjacency matrix". However, this intrinsic structural understanding is diluted by the attention sink. We theoretically formalize this dilution as a representation bottleneck, stemming from a fundamental conflict: the model's anisotropic bias, essential for language tasks, suppresses the topology-aware local aggregation required for graph reasoning. To address this, we propose a training-free solution, named StructuraL Attention SHarpening (SLASH), which amplifies this internal structural understanding via a plug-and-play attention redistribution. Experiments on pure graph tasks and molecular prediction validate that SLASH delivers significant and consistent performance gains across diverse LLMs.

图神经网络注意力机制大模型推理结构理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。