arXiv:2509.24260cs.AI2025-09被引 5

重新评估大模型图推理能力,发现聚焦算法设计比模仿更有效。

Rethinking and Benchmarking Large Language Models for Graph Reasoning

  • 引导大模型先设计算法再编码,突破传统模仿模式。
  • 新基准包含239个图问题、3041个实例,难度显著提升。
  • 简单新基线在多个任务上接近完美准确率,适合研究参考。

过去两年中,针对图推理的大语言模型(LLMs)研究广泛开展,旨在使模型理解图结构并解决各类图问题,其中图算法问题最为常见。尽管近期研究强调了大模型在图推理中的潜力,但实际表现仍不理想。本文指出现有方法与评测基准的问题,重新思考大模型图推理的发展方向。我们发现,基础模型如GPT-4o-mini被严重低估,因其推理焦点错误地集中在复现图算法而非设计算法。当将推理重点从复现转向设计时,基础模型能轻松解决大多数现有基准任务。为真正评估大模型的图推理能力,我们构建了一个更具挑战性的GraphAlgorithm基准,涵盖239个不同图问题和3041个测试实例,数据来自4个竞赛平台。最后,提出简单而强大的基线Simple-Reasoning-Then-Coding(Simple-RTC),引导模型先设计图算法再编码。该方法在现有基准上达到近似完美准确率,并在GraphAlgorithm基准上显著优于GPT-4o-mini及所有先前方法。这一强基线推动未来大模型图推理研究进一步发展。

原文摘要 · Abstract (English)

Large Language Models (LLMs) for Graph Reasoning have been extensively studied over the past two years, involving enabling LLMs to understand graph structures and reason on graphs to solve various graph problems, with graph algorithm problems being the most prevalent. Recent studies underscore the potential of LLMs in handling graph reasoning tasks, but their performance is underwhelming. In this work, we point out issues with existing methods and benchmarks, and rethink the direction that LLMs for graph reasoning should strive toward. We find that base models, e.g., GPT-4o-mini, are largely underestimated due to improper reasoning focus. Base models with reasoning focus redirected from replicating graph algorithms to designing them can easily solve most graph reasoning tasks in existing benchmarks. To truly evaluate the graph reasoning capabilities of LLMs, we construct a more challenging GraphAlgorithm benchmark, comprising 239 different graph problems and 3,041 test instances collected from 4 competition platforms. Finally, we introduce a simple and strong baseline Simple-Reasoning-Then-Coding (Simple-RTC)-which guides LLMs to design graph algorithms first and then code to address graph reasoning tasks. Simple-RTC achieves near-perfect accuracy on existing benchmarks and significantly outperforms GPT-4o-mini and all prior methods on the GraphAlgorithm benchmark. This strong baseline encourages further advancements in LLMs for Graph Reasoning in the future.

图推理大模型算法设计基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。