解决LLM处理图数据时的注意力错位问题,提升推理准确率。
Formalizing and Mitigating Structural Distortion in LLM Attention for Graph Reasoning
- 通过分析旋转位置编码机制,揭示图线性化导致注意力衰减的原理。
- 提出GaLA方法,在不增加计算负担下显著提升图推理性能。
- 适合关注图神经网络与大模型融合的研究者和实践者。
大型语言模型(LLMs)在文本属性图(TAGs)推理上展现出潜力。然而,将图结构线性化为序列会引入源于图带宽问题的结构扭曲。尽管已有研究指出这种扭曲会降低性能,但通常归因于提示设计或模型规模,其内在机制尚不清晰。本文揭示了旋转位置编码如何使图线性化产生依赖带宽的注意力衰减,抑制了在序列中被拉远的图相邻节点间的注意力。这一发现促使我们提出轻量级、推理时可应用的图对齐语言注意力(GaLA)方法,引导注意力聚焦于图相邻节点,同时保持语言模型的序列归纳偏置。在多个TAG基准测试中,GaLA以极低开销显著提升性能,表明结构扭曲是可纠正的瓶颈。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown promise for reasoning over Text-Attributed Graphs (TAGs). However, applying LLMs to graphs requires linearizing their structure into sequences, introducing distortion rooted in the graph bandwidth problem. While this distortion has been shown to degrade performance, it is often attributed to prompt design or model scale, leaving the underlying mechanism unclear. In this work, we show \textit{how} rotary positional embeddings turn graph linearization into bandwidth-dependent attention decay, suppressing attention between graph-adjacent nodes that are forced far apart in the serialized sequence. This shifts the focus of LLM-based graph reasoning from prompt engineering and scaling toward correcting attention misalignment. Motivated by this analysis, we propose \textbf{G}raph-\textbf{a}ligned \textbf{L}anguage \textbf{A}ttention (\textbf{GaLA}), a lightweight, inference-time modification for LLMs. GaLA biases attention toward graph-adjacent nodes while preserving the LLM's sequential inductive biases. Across TAG benchmarks, GaLA improves performance with negligible overhead, demonstrating that distortion is a correctable bottleneck in LLM-based graph reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。