通过细粒度评估推理步骤，提升大模型逻辑能力。
What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation
- 用因果分步评估法逐条检验推理相关性与连贯性
- 在两个专家标注数据集上验证效果，显著提升任务准确率
- 适合研究模型推理机制或优化训练数据的开发者
评估大语言模型（LLMs）通常只关注最终答案正确性，这种粗粒度方法忽视了推理过程质量。本文提出将推理质量分解为相关性（是否基于问题）和连贯性（是否逻辑自洽）两个维度，并引入因果分步评估（CaSE）方法，仅使用前序上下文评估每一步，避免事后偏差。我们在新构建的专家标注基准MRa-GSM8K和MRa-MATH上验证了该方法的有效性。更重要的是，使用CaSE评估结果筛选训练数据，可直接提升最终任务表现。本工作提供了一个可扩展的分析、调试与优化框架，证明超越简单正确性检查的实际价值。
原文摘要 · Abstract (English)
Evaluating large language models (LLMs) on final-answer correctness is the dominant paradigm. This approach, however, provides a coarse signal for model improvement and overlooks the quality of the underlying reasoning process. We argue that a more granular evaluation of reasoning offers a more effective path to building robust models. We decompose reasoning quality into two dimensions: relevance and coherence. Relevance measures if a step is grounded in the problem; coherence measures if it follows logically from prior steps. To measure these aspects reliably, we introduce causal stepwise evaluation (CaSE). This method assesses each reasoning step using only its preceding context, which avoids hindsight bias. We validate CaSE against human judgments on our new expert-annotated benchmarks, MRa-GSM8K and MRa-MATH. More importantly, we show that curating training data with CaSE-evaluated relevance and coherence directly improves final task performance. Our work provides a scalable framework for analyzing, debugging, and improving LLM reasoning, demonstrating the practical value of moving beyond validity checks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。