构建首个多维统一图推理基准，全面评测大模型的复杂推理能力。
Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models
- 基于大模型自动生成五维复杂图数据，半自动构造高质量测试集。
- 包含202个任务，揭示现有模型在代码与文本推理中的真实短板。
- 适合研究模型泛化性、推理增强方法的开发者使用。
图推理为评估大语言模型（LLMs）的推理能力提供了有前景的测试平台，因为图实例可程序化生成、结构可控，并可自然扩展至长输入场景。然而，现有图推理基准在数据复杂度覆盖上有限，高度依赖人工构建，且缺乏对文本与代码推理模式的统一评估。为此，我们提出{ extit{dataset}}，一个五阶段的半自动框架，用于构建复杂图推理基准。该框架从五个维度扩展基准覆盖范围：图规模、任务复杂度、任务描述、图加载方式和任务来源。框架利用大模型生成器自动生成任务描述、图数据、参考解、图加载脚本、问题形式及评估脚本，同时在关键质量控制环节保留人工验证。基于此框架，我们构建了一个包含202个任务的基准，并在文本、代码及增强推理设置下评估了多个大模型。实验表明，复杂度维度揭示了现有基准中不易察觉的模型局限性；已有微调模型在GraphGym任务上泛化能力差，而检索增强方法表现出情境依赖性，虽提升文本推理，但未一致改善编码推理。这些发现表明，该基准具有挑战性和诊断价值，为未来改进方法提供实证指导。代码与数据集将很快发布。
原文摘要 · Abstract (English)
Graph reasoning provides a promising testbed for evaluating the reasoning ability of large language models (LLMs), as graph instances can be programmatically generated, structurally controlled, and naturally scaled to long-input settings. However, existing graph reasoning benchmarks have limited coverage of data complexity, rely heavily on manual construction, and lack unified evaluation across text-based and code-based reasoning modes. To address these limitations, we propose {\dataset}, a five-stage \textit{semi-automatic} framework for constructing complex graph reasoning benchmarks. It expands benchmark coverage along five dimensions: \textit{Graph Size}, \textit{Task Complexity}, \textit{Task Description}, \textit{Graph Loading}, and \textit{Task Source}. The framework uses an LLM-based data generator to automatically produce task descriptions, graph data, reference solutions, graph-loading scripts, question forms, and evaluation scripts, while retaining human validation at key quality-control stages. Based on it, we construct a benchmark with $202$ tasks and evaluate LLMs under text-based, code-based, and augmented reasoning settings. Experiments show that the complexity dimensions reveal model limitations that are less visible in existing benchmarks; existing fine-tuned models struggle to generalize to GraphGym, whereas retrieval-augmented methods show scenario-dependent adaptability, improving textual reasoning but not consistently improving coding reasoning. These findings suggest that ours serves as a challenging and diagnostic benchmark for graph reasoning and provides empirical guidance for future enhancement methods. Code and dataset will be published soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。