arXiv:2504.20771cs.CL2025-04

用图灵机思想设计新评测基准,衡量大模型的规则遵循与状态管理能力。

Computational Reasoning of Large Language Models

  • 基于图灵机抽象,构建无领域依赖的多步推理评测任务
  • 在主流大模型上生成清晰的分步准确率曲线,反映推理能力
  • 适合评估模型基础推理能力,尤其关注规则执行与状态维护

随着大语言模型(LLMs)的快速发展与广泛应用,多维度评估变得愈发关键。然而现有评估常局限于特定领域且过于复杂,难以作为跨任务核心能力的有效代理。为此,我们提出一种理想代理任务应聚焦于可泛化的基础能力,且不依赖领域知识。图灵机提供了一种强大的理论视角,将复杂过程简化为基本、领域无关的计算操作,从而为评估广泛任务所需的基础计算能力提供了原则性框架。受此启发,我们引入 extbf{Turing Machine Bench}(TMBench),用于评估 LLM 在多步推理中严格遵循规则和准确管理内部状态的能力。TMBench 具备四大特点:自包含、知识无关的推理、极简的多步结构、可控难度以及基于图灵机的坚实理论基础。实验证明,TMBench 能有效评估计算推理能力,生成清晰的分步准确率曲线,揭示模型执行多步推理的过程能力。通过对比 TMBench 与现有推理基准的表现趋势,发现其与真实任务有强相关性,实现了真实任务评估与基础能力测评的桥梁作用。这些结果表明,TMBench 潜力作为跨领域推理评估的统一维度。代码与数据已公开于仓库。

原文摘要 · Abstract (English)

With the rapid development and widespread application of Large Language Models (LLMs), multidimensional evaluation has become increasingly critical. However, current evaluations are often domain-specific and overly complex, limiting their effectiveness as cross-domain proxies for core capabilities. To address these limitations and enable a unified and simple evaluation framework, an ideal proxy task should target a basic capability that generalizes across tasks and is independent of domain-specific knowledge. Turing machine provides a powerful theoretical lens by reducing complex processes to basic, domain-agnostic computational operations. This perspective offers a principled framework for evaluating basic computational abilities essential to a wide range of tasks. Motivated by this abstraction, we introduce \textbf{Turing Machine Bench}, a benchmark designed to assess the ability of LLMs to \textbf{strictly follow rules} and \textbf{accurately manage internal states} for multi-step, referred to as \textbf{computational reasoning}. TMBench incorporates four key features: self-contained and knowledge-agnostic reasoning, a minimalistic multi-step structure, controllable difficulty, and a solid theoretical foundation based on Turing machine. Empirical results demonstrate that TMBench serves as an effective proxy for evaluating computational reasoning on representative LLMs. It produces clear step-wise accuracy curves, revealing LLMs' ability to execute multi-step reasoning processes. By analyzing performance trends across TMBench and established reasoning benchmarks, we find strong correlations with real-world tasks, bridging real-task evaluation with basic ability assessment. These findings suggest that TMBench holds potential as a cross-domain dimension for evaluating reasoning in LLMs. Code and data are available at \href{https://github.com/HaitaoWuTJU/Turing-Machine-Bench}{Repo}.

大模型评测推理能力图灵机

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。