用可提取的符号程序评估模型数学推理一致性，发现顶级模型在变化输入下表现脆弱。
ReasonAgain: Using Extractable Symbolic Programs to Evaluate Mathematical Reasoning
- 从GSM8K和MATH数据集提取可执行符号程序作为评价基准。
- 新生成的问题使主流LLMs准确率显著下降，暴露推理不稳定性。
- 适合研究模型可泛化推理能力或评测数学问答系统的学者。
现有数学数据集通过最终答案或静态推理步骤评估大语言模型的推理能力，前者难以发现模型捷径和错误推理，后者难以容纳多种解法。本文提出使用符号程序作为自动化评估手段，检验模型在不同输入下是否能持续输出正确答案。我们利用GPT4-o从GSM8K和MATH数据集中提取可执行符号程序，并通过原始输入-输出对验证其正确性，发现这些程序确实封装了解题所需的核心推理逻辑。随后，我们基于提取的程序生成新的问题，构建测试集，并用于评估多个主流大模型。实验显示，相比原始静态样本，模型在新测试集上的准确率出现显著下降，表明当前顶尖大模型在数学推理上的脆弱性。
原文摘要 · Abstract (English)
Existing math datasets evaluate the reasoning abilities of large language models (LLMs) by either using the final answer or the intermediate reasoning steps derived from static examples. However, the former approach fails to surface model's uses of shortcuts and wrong reasoning while the later poses challenges in accommodating alternative solutions. In this work, we seek to use symbolic programs as a means for automated evaluation if a model can consistently produce correct final answers across various inputs to the program. We begin by extracting programs for popular math datasets (GSM8K and MATH) using GPT4-o. For those executable programs verified using the original input-output pairs, they are found to encapsulate the proper reasoning required to solve the original text questions. We then prompt GPT4-o to generate new questions using alternative input-output pairs based the extracted program. We apply the resulting datasets to evaluate a collection of LLMs. In our experiments, we observe significant accuracy drops using our proposed evaluation compared with original static examples, suggesting the fragility of math reasoning in state-of-the-art LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。