评估大模型推理需看中间步骤,而非仅看答案对错。
Measuring AI Reasoning: A Guide for Researchers

- 用可变深度搜索机制衡量推理能力
- 单次前向传播无法实现复杂推理过程
- 适合研究模型内部逻辑与可解释性的学者
本文为研究人员提供评估语言模型推理能力的指南,主张推理应通过适应性、多步搜索的证据来衡量,而非仅依赖最终答案正确率。在以评估为导向的定义下,推理需根据输入条件选择中间步骤并决定停止时机,我们将其形式化为类似搜索的过程。我们证明,大规模架构中的单次前向传播在实现可变深度计算方面存在结构性局限,因此建议采用中间解码和外部化推理轨迹作为合适的评估接口。核心观点是:仅凭最终答案准确率无法有效诊断或调试前沿模型生成解题过程的内在机制。因此,我们呼吁转向基于过程的评估,将中间推理轨迹的忠实性与有效性作为首要评估目标。
原文摘要 · Abstract (English)
In this paper, we offer a guide for researchers on evaluating reasoning in language models, building the case that reasoning should be assessed through evidence of adaptive, multi-step search rather than final-answer accuracy alone. Under an evaluation-oriented definition, reasoning requires selecting intermediate steps and halting according to input-dependent conditions, which we formalize as a search-like procedure. We show that single forward passes in scalable architectures are structurally limited in their ability to realize such variable-depth computation, motivating intermediate decoding and externalized reasoning traces as appropriate evaluation interfaces. Central to our argument is that final-answer accuracy alone is an insufficient measure of reasoning, because it provides little ability to diagnose or debug the underlying processes that produce individual solutions in frontier models. We therefore argue for a shift toward process-based evaluation, in which reasoning is assessed through the faithfulness and validity of intermediate reasoning traces as first-class evaluation targets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。