用树状思维链提升小模型推理能力，减少长思考中的幻觉。
Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models
- 通过蒙特卡洛树搜索构建树状思维链数据
- 新方法使小模型在数学、规划等任务上表现显著提升
- 适合需要高效推理的小模型部署场景
大型推理模型（LRMs）如OpenAI o1和DeepSeek-R1通过增加测试时计算量和生成长链式思维（CoT）展现了强大推理能力。知识蒸馏——在LRM生成的数据上进行后训练——是提升小型模型推理能力的简单有效方法，但存在关键瓶颈：我们发现，蒸馏所得的长CoT数据对小型模型学习困难，并在使用监督微调（SFT）和强化学习（RL）时导致过思考等偏差。为此，我们提出从头构建基于蒙特卡洛树搜索（MCTS）的树状CoT数据，并采用一系列思维链感知方法，包括思维长度平衡、细粒度DPO和联合后训练目标，以增强SFT与RL效果。在GSM8K、MATH、AIME等数学基准，Multi-IF指令跟随，以及Blocksworld规划任务上的评估表明，相比标准蒸馏方法，我们的方案显著提升了小型模型的推理性能，同时减少了长时间思考中的幻觉。项目主页：https://github.com/AIDC-AI/Marco-o1。
原文摘要 · Abstract (English)
Large Reasoning Models(LRMs) such as OpenAI o1 and DeepSeek-R1 have shown remarkable reasoning capabilities by scaling test-time compute and generating long Chain-of-Thought(CoT). Distillation--post-training on LRMs-generated data--is a straightforward yet effective method to enhance the reasoning abilities of smaller models, but faces a critical bottleneck: we found that distilled long CoT data poses learning difficulty for small models and leads to the inheritance of biases (i.e. over-thinking) when using Supervised Fine-tuning (SFT) and Reinforcement Learning (RL) methods. To alleviate this bottleneck, we propose constructing tree-based CoT data from scratch via Monte Carlo Tree Search(MCTS). We then exploit a set of CoT-aware approaches, including Thoughts Length Balance, Fine-grained DPO, and Joint Post-training Objective, to enhance SFT and RL on the constructed data. We conduct evaluation on various benchmarks such as math (GSM8K, MATH, AIME). instruction-following (Multi-IF) and planning (Blocksworld), results demonstrate our approaches substantially improve the reasoning performance of distilled models compared to standard distilled models via reducing the hallucinations in long-time thinking. The project homepage is https://github.com/AIDC-AI/Marco-o1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。