arXiv:2601.04996cs.AI2026-01ACL被引 1

测试大模型对算法的理解程度,发现其在复杂算法上表现差。

AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms?

  • 构建3000+道算法题库,按优化类型分类评估模型
  • 模型在动态规划等全局优化算法上准确率仅49%
  • 揭示模型因低熵标记过早放弃正确解法的缺陷

推理能力已成为大型推理模型(LRMs)发展的核心焦点。尽管在MATH500和LiveCodeBench等基准上取得显著进展,现有算法推理评测仍显不足,无法回答关键问题:LRMs是否真正掌握算法推理?为此,我们提出AlgBench,一个由计算机科学专家精心设计的算法中心型评测基准。AlgBench包含超过3,000道原创题目,覆盖27种算法,按欧几里得结构、非欧几里得结构、非优化、局部优化、全局优化和启发式优化等类别组织。对领先模型(如Gemini-3-Pro、DeepSeek-v3.2-Special and GPT-o3)的实证评估显示显著性能差异:模型在非优化任务上表现良好(最高达92%),但在全局优化算法(如动态规划)上准确率骤降至约49%。进一步分析揭示了‘策略性提前偏离’现象:模型因必要低熵标记而过早放弃正确算法设计。这些发现暴露了以问题为中心的强化学习的根本局限,并强调了算法中心训练范式的必要性。

原文摘要 · Abstract (English)

Reasoning ability has become a central focus in the advancement of Large Reasoning Models (LRMs). Although notable progress has been achieved on several reasoning benchmarks such as MATH500 and LiveCodeBench, existing benchmarks for algorithmic reasoning remain limited, failing to answer a critical question: Do LRMs truly master algorithmic reasoning? To answer this question, we propose AlgBench, an expert-curated benchmark that evaluates LRMs under an algorithm-centric paradigm. AlgBench consists of over 3,000 original problems spanning 27 algorithms, constructed by ACM algorithmic experts and organized under a comprehensive taxonomy, including Euclidean-structured, non-Euclidean-structured, non-optimized, local-optimized, global-optimized, and heuristic-optimized categories. Empirical evaluations on leading LRMs (e.g., Gemini-3-Pro, DeepSeek-v3.2-Speciale and GPT-o3) reveal substantial performance heterogeneity: while models perform well on non-optimized tasks (up to 92%), accuracy drops sharply to around 49% on globally optimized algorithms such as dynamic programming. Further analysis uncovers \textbf{strategic over-shifts}, wherein models prematurely abandon correct algorithmic designs due to necessary low-entropy tokens. These findings expose fundamental limitations of problem-centric reinforcement learning and highlight the necessity of an algorithm-centric training paradigm for robust algorithmic reasoning.

算法理解大模型评测动态规划推理能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。