诊断图增强大模型如何真正使用图数据,发现仅给图信息不够,需设计专用接口。
Graph Evidence Is Not Enough: Diagnosing Native Decoder Use in Graph-Augmented LLMs
- 构建三重对照实验,分离图信息可用性、可读性和解码器理解能力
- 提出S²GE接口,使模型在多个数据集上准确率提升超50个百分点
- 揭示图数据混乱时模型表现恶化,证明结构设计至关重要
图增强型大语言模型通常假设外部计算生成的图证据可被原生解码器直接使用。我们通过一个限定任务HopQA进行诊断,要求预测两个查询节点间的最短路径距离。由于答案为小整数且目标纯粹是拓扑结构,失败无法归因于开放生成或评估模糊。然而现有基线仍在此任务中表现不佳,表明提供图证据不等于使其可用。为此,我们提出干预三角:可读图证据、打乱图证据和无图输入三种匹配条件,以区分证据引入、结构可读性与解码器可用拓扑。基于此诊断,我们提出S²GE,采用查询感知采样、端点与邻近性排序及结构保持对齐。在DBLP、Biomedical、GoodReads和PubMed上,S²GE分别取得36.5%、57.8%、76.6%和52.0%的严格精确匹配分数,平均优于最强原生生成基线53.5个百分点。干预分析进一步揭示了有害打乱、打乱鲁棒和无图饱和三种模型行为模式。
原文摘要 · Abstract (English)
Graph-augmented large language models often assume that graph evidence produced by external computation and placed in the input can be used by the native decoder. We test this assumption with HopQA, a deliberately bounded diagnostic that asks for the shortest-hop distance between two query nodes. Because the answer is a small integer and the target is purely topological, failure cannot be dismissed as open-ended generation or ambiguous evaluation. Yet existing graph-augmented baselines still fail on this setting, showing that providing graph evidence is not the same as making it usable. We introduce an intervention triangle with three matched conditions: readable graph evidence, shuffled graph evidence, and no-graph input. This separates evidence inclusion, structural readability, and decoder-usable topology. Guided by this diagnosis, we present S$^2$GE as an instance showing that diagnosis-driven interface design can improve native decoder usability. S$^2$GE uses query-aware sampling, endpoint and proximity-based ordering, and structure-preserving alignment. Across DBLP, Biomedical, GoodReads, and PubMed, S$^2$GE achieves strict exact-match scores of $36.5\%$, $57.8\%$, $76.6\%$, and $52.0\%$, improving over the strongest native-generation baseline by $53.5$ points on average. The interventions further reveal harmful-shuffle, shuffle-robust, and no-graph-saturated regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。