大模型解数学题时，代码执行反而不如纯推理稳健。
Reasoning, Code, or Both? How Large Language Models Handle Variations in Math Questions
- 对比三种解题方式：自然语言推理、单次代码生成、逐步编码。
- 纯推理准确率下降仅1.3个百分点，代码方法下降超1.7个百分点。
- 适合关注模型鲁棒性的研究者，尤其在教育场景应用时需谨慎使用代码方法。
大型语言模型（LLMs）在数学推理基准上表现优异，但当问题通过简单更改名称或数字进行扰动时，性能显著下降。代码执行方法（如PAL和SBSC）被提出作为提升准确性的方案，但其对推理鲁棒性的影响尚未系统评估。本研究在GSM-Symbolic数据集的1000个问题上，比较了三种方法：链式思维（CoT）提示、单次代码执行（PAL）和逐步编码（SBSC），均在Claude Haiku 4.5上运行。结果表明，纯推理方法最稳健，准确率下降仅1.3个百分点，1.8%的问题在扰动下失效；而PAL下降1.7个百分点，3.1%的问题失效，SBSC介于两者之间。尽管差异未达统计显著性（p = .096），但趋势一致，说明代码执行并未提升对小学级题目变化的鲁棒性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) achieve impressive accuracy on mathematical reasoning benchmarks, yet their performance drops when problems are modified with simple changes like different names or numbers. Code execution methods, which let models generate and run Python code instead of reasoning in natural language, have been proposed as a solution, but their effect on reasoning robustness (the ability to maintain accuracy across problem variations) has not been systematically tested. This study evaluates three approaches on 1,000 problems from the GSM-Symbolic dataset: pure reasoning using chain-of-thought (CoT) prompting, single-shot code execution using Program-Aided Language models (PAL), and iterative code execution using Step-by-Step Coding (SBSC). All three were run on paired original and modified problems using Claude Haiku 4.5. CoT was the most robust method, with an accuracy drop of 1.3 percentage points and 1.8% of problems breaking under perturbation. PAL was the least robust at 1.7 percentage points and 3.1% broke, with SBSC falling in between. Although these differences were not statistically significant ($p = .096$), the directional trend was consistent across all measures, suggesting that code execution, whether single-shot or iterative, does not improve reasoning robustness on grade-school-level problem variations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。