用蒙特卡洛树搜索自动生成推理过程数据，提升大模型数学推理能力。
Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
- 让大模型自己生成推理步骤并评分，通过加权损失训练增强推理。
- 在两个数学推理数据集上性能显著提升，且能力可跨数据集迁移。
- 适合研究大模型推理机制与自我监督训练方法的学者参考。
大型语言模型（LLMs）在多种任务中展现出卓越能力，但推理仍是其短板。为提升LLM的推理能力，过程监督比结果监督更有效。本文提出利用蒙特卡洛树搜索（MCTS）让LLM自身生成过程监督数据用于训练。具体地，用LLM采样推理步骤，并为其分配反映“相对正确性”的分数，随后通过最小化生成步骤的加权对数似然来训练模型。该生成-训练流程迭代进行直至收敛。实验表明，所提方法显著提升了LLM在两个数学推理数据集上的表现，且在某一数据集上训练的模型在另一数据集上也取得更好效果，证明了增强推理能力的可迁移性。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated their remarkable capacity across a variety of tasks. However, reasoning remains a challenge for LLMs. To improve LLMs' reasoning ability, process supervision has proven to be better than outcome supervision. In this work, we study using Monte Carlo Tree Search (MCTS) to generate process supervision data with LLMs themselves for training them. We sample reasoning steps with an LLM and assign each step a score that captures its "relative correctness," and the LLM is then trained by minimizing weighted log-likelihood of generating the reasoning steps. This generate-then-train process is repeated iteratively until convergence.Our experimental results demonstrate that the proposed methods considerably improve the performance of LLMs on two mathematical reasoning datasets. Furthermore, models trained on one dataset also exhibit improved performance on the other, showing the transferability of the enhanced reasoning ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。