通过优化推理路径提升大模型解题能力，避免错误发散。
Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths
- 在每步推理中奖励正确路径，惩罚错误分支。
- GSM8K和MMLU(STEM)分别提升3.1%和4.3%。
- 无需人工标注，适合大规模推理任务优化。
OpenAI o1等先进模型虽具备出色的一致性推理能力，但在复杂问题上仍易出错，导致推理路径偏离。我们归因于解题空间过大，每一步都可能产生偏差。为此提出推理路径优化（RPO）框架，通过鼓励有利分支、惩罚不利分支，提升模型整体解题性能。RPO不依赖大规模人工标注或闭源模型输出，具有高可扩展性和数据效率。实验聚焦多步推理任务，如数学应用题和科学考试题，在GSM8K和MMLU（STEM）上分别实现最高3.1%和4.3%的性能提升。代码与数据见https://reasoning-paths.github.io。
原文摘要 · Abstract (English)
Advanced models such as OpenAI o1 exhibit impressive problem-solving capabilities through step-by-step reasoning. However, they may still falter on more complex problems, making errors that disrupt their reasoning paths. We attribute this to the expansive solution space, where each step has the risk of diverging into mistakes. To enhance language model reasoning, we introduce a specialized training framework called Reasoning Paths Optimization (RPO), which enables learning to reason and explore from diverse paths. Our approach encourages favorable branches at each reasoning step while penalizing unfavorable ones, enhancing the model's overall problem-solving performance. Reasoning Paths Optimization does not rely on large-scale human-annotated rationales or outputs from closed-source models, making it scalable and data-efficient. We focus on multi-step reasoning tasks, such as math word problems and science-based exam questions. The experiments demonstrate that our framework significantly enhances the reasoning performance of large language models, with up to 3.1% and 4.3% improvement on GSM8K and MMLU (STEM) respectively. Our data and code can be found at https://reasoning-paths.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。