测试大模型对数学解题步骤的鲁棒性，发现其易受非标准但正确解法干扰。
Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification
- 设计线性方程基准，评估模型对解题步骤和错误定位的能力。
- 非标准解法下误判率高达75.6%-85.3%，显著低于标准解法表现。
- 验证能力需独立于求解能力衡量，尤其在有精确答案的简单领域。
大型语言模型（LLMs）被越来越多地用作评分器、验证者和过程审计员，但现有数学评估仍主要关注最终答案的准确性，这可能掩盖了模型对非标准但有效的解题过程的判断能力。本文提出一个控制性线性方程基准，用于评估模型在验证者角色中的表现。每个任务要求模型判断最终答案是否正确、每一步解题过程是否正确，并指出第一个错误步骤。对GPT-OSS 20B、Qwen3-14B和Phi-4-Reasoning等主流开源大模型的评估显示，存在显著的鲁棒性差距：能准确评价标准解法的模型，在面对逻辑等价但形式不同的扰动解法时表现大幅下降。基础模型在标准解法上表现良好，但在扰动解法上性能显著退化，尤其在错误定位方面。对于有效扰动解法，基础模型的误拒率高达75.6%-85.3%，显示出对标准解法形式的高度敏感。监督微调、知识蒸馏和测试时计算虽在某些场景提升鲁棒性，但效果依赖模型且可能以牺牲标准性能为代价。结果表明，可靠的流程级验证仍具挑战性，验证鲁棒性应与求解准确性分开评估，即使在具有精确真值的简单代数领域亦如此。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used as graders, verifiers, and process auditors, but most mathematical evaluations still emphasize final-answer accuracy. This can obscure whether a model can verify a non-canonical but valid solution trace. We introduce a controlled linear-equation benchmark for evaluating LLMs in the evaluator role. Each instance asks the model to judge final-answer correctness, step-level trace correctness, and the first incorrect step. Our evaluation of state-of-the-art open LLMs reveals a significant robustness gap: models that accurately evaluate canonical solutions often fail when presented with perturbed but logically equivalent variants. Across GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning, base models perform well on canonical traces but degrade substantially on perturbed traces, especially for error localization. On valid perturbed traces, base-model false-rejection rates reach 75.6-85.3%, showing strong sensitivity to canonical solution form. Supervised fine-tuning, distillation, and test-time compute improve robustness in some settings, but gains are model dependent and can trade off against canonical performance. The results show that reliable process-level verification remains challenging, and evaluator robustness should be measured separately from solver accuracy, even in a simple algebraic domain with exact ground truth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。