评测大模型在图数据上进行复杂推理的能力，发现现有模型普遍表现不佳。
GraphInfer-Bench: Benchmarking LLM's Inference Capability on Graphs
- 构建五类图推理任务，答案需综合节点与邻域信息，无法从单点或路径获取。
- 包含4.2万条样本，涵盖六种真实图数据，质量经四层自动筛选保障。
- 大模型在比较类任务中表现差，传统GNN反而在社区检测等任务上更优。
图分析支撑诸多应用，其答案无法通过单一节点或路径获取：如洗钱网络、药物重定位、用户偏好和科学主题等，均需结合节点及其邻域信息推断。本文提出GraphInfer-Bench，一个用于评测大模型图推理能力的基准：生成开放性答案，且该答案无法由任一单节点或路径检索获得。现有图问答协议（如算法模拟、节点分类、单节点描述、知识图谱问答、GraphRAG）均无法测试此能力。GraphInfer-Bench设计五类任务，涵盖描述（某区域特征）与比较（区域间差异），每项任务的正确答案均不依赖单一节点。基准包含42,000个样本，覆盖六个真实世界图数据集，通过自动化生成与四层质量控制流程确保可靠性。我们评估了四类方法：图-标记对齐模型、零样本前沿闭源大模型、Graph2Text监督微调模型及普通GNN作为结构参照。结果表明，无一类方法能全面覆盖所有任务。图-标记对齐模型在描述任务（关系、主题）上表现尚可，但在比较任务上完全失效；前沿大模型在异常检测与社区划分上领先，但对掩码节点预测表现滞后；Graph2Text SFT在描述任务中最强，仍落后于前沿大模型在比较任务上的表现。在所有任务中，普通GNN均达到或超过最强的大模型表现，尤其在社区检测任务上差距最大。结果揭示：图推理是当前大模型存在的开放性能力鸿沟，而非特定架构固有属性。
原文摘要 · Abstract (English)
Graph analysis underlies many applications whose answers cannot be looked up in a single record or retrieved along a path: laundering rings, drug repurposing, user preference, and scientific theme are all inferred from a node together with its neighbourhood. We introduce GraphInfer-Bench, a benchmark for whether LLMs can perform this graph inference: producing an open-ended answer that no single node supports and no path retrieves. Existing graph-QA protocols cannot test this capability: algorithm simulation, node classification, single-node description, KG-QA, and GraphRAG all admit answers retrievable from one node or along a path. GraphInfer-Bench defines five tasks along Description (what a region is) and Comparison (how regions differ), each constructed so the ground truth lives in no single node. The release contains 42,000 samples across six real-world graphs, produced automatically and screened by a four-layer quality-control protocol. We evaluate four method families against the same tasks: graph-token alignment models, zero-shot frontier closed-source LLMs, Graph2Text supervised fine-tuning, and plain GNNs as a structural reference. No method family closes the gap. Graph-token alignment partially handles description tasks (relational, theme) but collapses on comparison tasks. Frontier LLMs lead on outlier detection and community partition among LLM-based methods but lag on masked-node prediction. Graph2Text SFT is the strongest LLM-based method on the description side yet falls behind frontier LLMs on comparison. Across every task, plain GNNs match or beat the strongest LLM-based row, with the largest margin on community detection. GraphInfer-Bench surfaces graph inference as an open capability gap rather than a property of any one architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。