用图算法题暴露大模型推理短板，发现长文本下准确率跌破50%。
Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems
- 设计图算法基准测试，支持长上下文、可程序验证
- 120节点以上图时准确率低于50%，因记忆弱与冗余推理
- 揭示模型过度自检却无效，适合研究推理机制的学者
大型推理模型（LRMs）发展迅速，但现有数学、编程和常识推理基准存在局限：缺乏长上下文评估、挑战性不足，且答案难以程序化验证。我们提出GrAlgoBench，一个基于图算法问题的基准测试，专门用于评估LRMs。这类问题具备长上下文推理需求、可精细调节难度，并支持标准化程序化评估。在九项任务中，系统实验揭示当前LRMs两大弱点：一是上下文长度增加时准确率急剧下降，当图超过120个节点时低于50%，主因是执行错误频发、记忆能力弱及冗余推理；二是存在过度思考现象，主要由大量但基本无效的自我验证导致，显著拉长推理路径却不提升正确率。通过暴露这些局限，GrAlgoBench确立了图算法问题作为严谨、多维度且实际相关的推理研究测试平台。代码开源于https://github.com/Bklight999/GrAlgoBench。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have advanced rapidly; however, existing benchmarks in mathematics, code, and common-sense reasoning remain limited. They lack long-context evaluation, offer insufficient challenge, and provide answers that are difficult to verify programmatically. We introduce GrAlgoBench, a benchmark designed to evaluate LRMs through graph algorithm problems. Such problems are particularly well suited for probing reasoning abilities: they demand long-context reasoning, allow fine-grained control of difficulty levels, and enable standardized, programmatic evaluation. Across nine tasks, our systematic experiments reveal two major weaknesses of current LRMs. First, accuracy deteriorates sharply as context length increases, falling below 50% once graphs exceed 120 nodes. This degradation is driven by frequent execution errors, weak memory, and redundant reasoning. Second, LRMs suffer from an over-thinking phenomenon, primarily caused by extensive yet largely ineffective self-verification, which inflates reasoning traces without improving correctness. By exposing these limitations, GrAlgoBench establishes graph algorithm problems as a rigorous, multidimensional, and practically relevant testbed for advancing the study of reasoning in LRMs. Code is available at https://github.com/Bklight999/GrAlgoBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。