用程序验证推理路径，提升大模型数学解题准确率。
Not All Votes Count! Programs as Verifiers Improve Self-Consistency of Language Models for Math Reasoning
- 将自然语言解法转为程序，验证中间步骤正确性。
- 在GSM8K上最高提升18%，MATH-500提升8%。
- 适用于各类开源大模型，尤其适合数学推理任务。
大型语言模型（LLMs）在解决数学推理问题上表现出越来越强的能力，但许多开源LLMs在中间推理步骤中仍存在计算错误和语义理解偏差。本文提出Prove框架，通过将自然语言解法转换为程序，作为验证机制，在聚合最终答案前筛选掉不一致的推理路径。与原始多数投票不同，该方法仅聚合程序输出与生成解法一致的路径。我们在13个来自不同模型家族和规模（0.5B至13B参数）的开源LLMs上进行了广泛实验，覆盖八个数学基准测试。结果表明，Prove在所有模型规模和数据集上均优于传统多数投票，其中在GSM8K上最高提升18%，在MATH-500上提升8%。代码已开源：https://github.com/declare-lab/prove。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown increasing competence in solving mathematical reasoning problems. However, many open-source LLMs still struggle with errors in calculation and semantic understanding during intermediate reasoning steps. In this work, we introduce Prove, a simple yet effective framework that leverages translated programs derived from natural language solutions as a verification mechanism to filter out potentially incorrect reasoning paths before aggregating final answers. Unlike vanilla majority voting, our approach filters out solutions whose corresponding program output is inconsistent with the generated solution, aggregating only those that pass verification. We conducted extensive experiments using 13 open-source LLMs from various model families and sizes, ranging from 0.5B to 13B parameters, across eight mathematical benchmarks. Our results show that Prove consistently outperforms vanilla majority voting as a heuristic for solving mathematical reasoning tasks across all model sizes and datasets, achieving improvements of up to 18% on GSM8K and 8% on MATH-500. Our codes are available at https://github.com/declare-lab/prove.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。