用错误树分析数学解题过程，让AI评估更像人。
StepMathAgent: A Step-Wise Agent for Evaluating Mathematical Processes through Tree-of-Error
- 构建错误树追踪每步逻辑，精准定位解题漏洞
- 在1000个分步题目上超越现有方法，接近人类评分
- 适合评估证明题和开放问题，尤其适合教学场景
评估大语言模型的数学能力对全面衡量其性能至关重要。现有方法多仅关注最终答案，导致评估结果不准确且不可解释，难以评估证明题或开放性问题。为此，我们提出基于错误树的数学过程评估代理StepMathAgent，包含逻辑步骤分割、步骤评分、分数聚合与错误树生成四类内部核心操作，以及难度校准、简洁性评估、完整性验证和格式评估四类外部扩展模块。同时，我们构建了包含1000个分步解析实例的StepMathBench基准，数据源自200道高质量数学题，按题型、学科类别和难度分级。实验表明，StepMathAgent在该基准上优于所有现有方法，展现出与人类评估偏好一致的结果，并具备广泛适用性。代码与数据已开源。
原文摘要 · Abstract (English)
Evaluating mathematical capabilities is critical for assessing the overall performance of large language models (LLMs). However, existing evaluation methods often focus solely on final answers, resulting in highly inaccurate and uninterpretable evaluation outcomes, as well as their failure to assess proof or open-ended problems. To address these issues, we propose a novel mathematical process evaluation agent based on Tree-of-Error, called StepMathAgent. This agent incorporates four internal core operations: logical step segmentation, step scoring, score aggregation and error tree generation, along with four external extension modules: difficulty calibration, simplicity evaluation, completeness validation and format assessment. Furthermore, we introduce StepMathBench, a benchmark comprising 1,000 step-divided process evaluation instances, derived from 200 high-quality math problems grouped by problem type, subject category and difficulty level. Experiments on StepMathBench show that our proposed StepMathAgent outperforms all state-of-the-art methods, demonstrating human-aligned evaluation preferences and broad applicability to various scenarios. Our data and code are available at https://github.com/SHU-XUN/StepMathAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。