用MCTS轨迹训练大模型自进化，提升推理能力
Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learning
- 构建树中同父节点的步骤级轨迹对，挖掘深层推理信息
- 在数学推理任务上超越现有方法，显著增强模型推理能力
- 适合想提升大模型逻辑推理能力的研究者和开发者
蒙特卡洛树搜索（MCTS）近期成为提升大语言模型（LLMs）推理能力的强大工具。虽然监督微调（SFT）或直接偏好优化（DPO）已能从MCTS中提炼高质量行为，但现有蒸馏方法未能充分利用MCTS生成的丰富轨迹信息，限制了推理性能的进一步提升。本文提出AlphaLLM-CPL，一种新型成对训练框架，使LLMs通过MCTS行为蒸馏实现自我改进。该框架通过两项关键创新高效利用MCTS轨迹：(1) 从搜索树中同一父节点的子节点构建步骤级轨迹对，提供细粒度的推理过程信息；(2) 引入课程偏好学习，动态调整每轮离线训练中轨迹对的训练顺序，优先处理关键学习阶段，缓解过拟合。在数学推理任务上的实验表明，AlphaLLM-CPL显著优于先前的MCTS行为蒸馏方法，大幅提升了LLMs的推理能力。
原文摘要 · Abstract (English)
Monte Carlo Tree Search (MCTS) has recently emerged as a powerful technique for enhancing the reasoning capabilities of LLMs. Techniques such as SFT or DPO have enabled LLMs to distill high-quality behaviors from MCTS, improving their reasoning performance. However, existing distillation methods underutilize the rich trajectory information generated by MCTS, limiting the potential for improvements in LLM reasoning. In this paper, we propose AlphaLLM-CPL, a novel pairwise training framework that enables LLMs to self-improve through MCTS behavior distillation. AlphaLLM-CPL efficiently leverages MCTS trajectories via two key innovations: (1) AlphaLLM-CPL constructs stepwise trajectory pairs from child nodes sharing the same parent in the search tree, providing step-level information for more effective MCTS behavior distillation. (2) AlphaLLM-CPL introduces curriculum preference learning, dynamically adjusting the training sequence of trajectory pairs in each offline training epoch to prioritize critical learning steps and mitigate overfitting. Experimental results on mathematical reasoning tasks demonstrate that AlphaLLM-CPL significantly outperforms previous MCTS behavior distillation methods, substantially boosting the reasoning capabilities of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。