用最优优先搜索让大模型少废话，又准又快。
BFS-PO: Best-First Search for Large Reasoning Models
- 引入最优优先搜索策略，通过最大熵节点回溯找最短正确路径。
- 训练中逐步生成更短回答，准确率不降反升。
- 适合追求高效推理的部署场景，尤其适配长链推理模型。
大型推理模型（如 OpenAI o1 和 DeepSeek-R1）在长推理链任务中表现优异，但也带来计算成本激增和输出冗长的问题，即“过度思考”。这种现象常由强化学习算法（如 GRPO/DAPO）加剧。本文提出 BFS-PO，一种基于最优优先搜索探索策略的强化学习算法。BFS-PO 通过基于最大熵节点的回溯机制，寻找最短正确答案，使模型在训练中逐步生成更短的响应。在多个基准测试和不同基础模型上验证，BFS-PO 能同时提升准确率并缩短输出长度。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) such as OpenAI o1 and DeepSeek-R1 have shown excellent performance in reasoning tasks using long reasoning chains. However, this has also led to a significant increase of computational costs and the generation of verbose output, a phenomenon known as overthinking. The tendency to overthinking is often exacerbated by Reinforcement Learning (RL) algorithms such as GRPO/DAPO. In this paper, we propose BFS-PO, an RL algorithm which alleviates this problem using a Best-First Search exploration strategy. Specifically, BFS-PO looks for the shortest correct answer using a backtracking mechanism based on maximum entropy nodes. By generating progressively shorter responses during training, BFS-PO learns to produce concise reasoning chains. Using different benchmarks and base LRMs, we show that BFS-PO can simultaneously increase the LRM accuracy and shorten its answers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。