用树搜索思想提升大模型推理数据生成效率，更准更快。
FastMCTS: A Simple Sampling Strategy for Data Synthesis
- 借鉴蒙特卡洛树搜索，按步评估并动态优化采样路径。
- 生成相同数量token时，正确推理路径多出30%以上。
- 适合需要高质量推理数据的训练场景，轻量易部署。
合成高质量的多步推理数据能显著提升大语言模型在各类任务上的表现。然而，现有方法多依赖拒绝采样，独立生成轨迹，存在效率低下和不同难度问题间采样不均衡的问题。本文提出FastMCTS，一种受蒙特卡洛树搜索启发的数据合成策略。该方法提供逐步评估信号，实现更高效的采样，并促进不同难度问题间的平衡采样。在中英文推理数据集上的实验表明，随着生成标记数增加，FastMCTS生成的正确推理路径比拒绝采样多出30%以上。此外，在相近的合成数据预算下，基于FastMCTS生成数据训练的模型在多个基准上比拒绝采样训练的模型性能高3.9%。作为轻量级采样策略，FastMCTS为高质量推理数据合成提供了高效实用的替代方案。代码将很快开源。
原文摘要 · Abstract (English)
Synthetic high-quality multi-step reasoning data can significantly enhance the performance of large language models on various tasks. However, most existing methods rely on rejection sampling, which generates trajectories independently and suffers from inefficiency and imbalanced sampling across problems of varying difficulty. In this work, we introduce FastMCTS, an innovative data synthesis strategy inspired by Monte Carlo Tree Search. FastMCTS provides a more efficient sampling method for multi-step reasoning data, offering step-level evaluation signals and promoting balanced sampling across problems of different difficulty levels. Experiments on both English and Chinese reasoning datasets demonstrate that FastMCTS generates over 30\% more correct reasoning paths compared to rejection sampling as the number of generated tokens scales up. Furthermore, under comparable synthetic data budgets, models trained on FastMCTS-generated data outperform those trained on rejection sampling data by 3.9\% across multiple benchmarks. As a lightweight sampling strategy, FastMCTS offers a practical and efficient alternative for synthesizing high-quality reasoning data. Our code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。