用填空题训练大模型，让其更像人一样解数学题。
ClozeMath: Improving Mathematical Reasoning in Language Models by Learning to Fill Equations
- 用填空式任务预测被遮蔽的方程，模仿人类学数学方式。
- 在GSM8K等数据集上表现优于现有方法，提升推理准确率。
- 适合想提升模型数学推理能力的研究者和开发者。
大型语言模型（LLMs）通过训练反映人类思维过程的数据（如思维链格式）得到了能力提升。然而，有证据表明传统的下一个词预测机制可能无法充分捕捉人类学习思考的方式。受人类数学推理泛化方式的启发，我们提出一种名为ClozeMath的新方法，用于微调LLMs以增强数学推理能力。ClozeMath采用文本填空任务，从给定解法中预测被遮蔽的方程，类似于人类学习中的填空练习。在GSM8K、MATH和GSM-Symbolic数据集上的实验表明，该方法在使用束搜索（Beam Search）和思维链解码（Chain-of-Thought decoding）两种测试时缩放解码算法下，性能和鲁棒性均超过强基线方法Masked Thought。此外，我们还进行了消融实验，分析了不同架构与实现选择对方法效果的影响。
原文摘要 · Abstract (English)
The capabilities of large language models (LLMs) have been enhanced by training on data that reflects human thought processes, such as the Chain-of-Thought format. However, evidence suggests that the conventional scheme of next-word prediction may not fully capture how humans learn to think. Inspired by how humans generalize mathematical reasoning, we propose a new approach named ClozeMath to fine-tune LLMs for mathematical reasoning. Our ClozeMath involves a text-infilling task that predicts masked equations from a given solution, analogous to cloze exercises used in human learning. Experiments on GSM8K, MATH, and GSM-Symbolic show that ClozeMath surpasses the strong baseline Masked Thought in performance and robustness, with two test-time scaling decoding algorithms, Beam Search and Chain-of-Thought decoding. Additionally, we conduct an ablation study to analyze the effects of various architectural and implementation choices on our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。