让大模型用更短路径做出准确推理,关键在奖励设计与训练策略。
The Art of Efficient Reasoning: Data, Reward, and Optimization
- 分两阶段训练:先调长度,再优化思维质量。
- 长推理不等于高准确,需避免短答案即正确的陷阱。
- 方法通用性强,适配0.6B到30B不同规模模型。
大型语言模型(LLMs)在采用链式思考(CoT)推理时表现更好,但计算开销巨大。为提升效率,研究聚焦于激励模型生成简短而准确的推理路径,通常通过强化学习(RL)进行奖励重塑。本文系统研究了大模型高效推理的机制。为全面评估,提出细粒度指标,包括正确性条件下的长度分布,以及在2k至32k token预算范围内的性能表现。实验发现训练过程遵循两阶段范式:长度适应与推理精炼。基于约0.2百万GPU小时的统一协议实验,我们剖析了提示设计、采样轨迹、奖励塑造与优化策略。核心发现是保持足够密度的正向奖励信号,并避免‘短即正确’的误区。此外,学习到的长度偏好具有跨领域与难度的泛化能力。我们将这些发现提炼为实用指导,并在从0.6B到30B的Qwen3系列模型上验证其鲁棒性与普适性。权重已公开于https://wutaiqiang.github.io/project/Art。
原文摘要 · Abstract (English)
Large Language Models (LLMs) consistently benefit from scaled Chain-of-Thought (CoT) reasoning, but also suffer from heavy computational overhead. To address this issue, efficient reasoning aims to incentivize short yet accurate thinking trajectories, typically through reward shaping with Reinforcement Learning (RL). In this paper, we systematically investigate the mechanics of efficient reasoning for LLMs. For comprehensive evaluation, we advocate for more fine-grained metrics, including length distribution conditioned on correctness and performance across a wide spectrum of token budgets ranging from 2k to 32k. First, we reveal that the training process follows a two-stage paradigm: length adaptation and reasoning refinement. Through extensive experiments (about 0.2 million GPU hours) in a unified protocol, we deconstruct training prompts and rollouts, reward shaping, and optimization strategies. A central finding is to maintain a sufficient density of positive reward signals and avoid the short-is-correct trap. Moreover, the learned length bias generalizes across domains and difficulty levels. We distill these findings into valuable insights and practical guidelines, and validate them across the Qwen3 models ranging from 0.6B to 30B, demonstrating the robustness and generalization. Weights are available at https://wutaiqiang.github.io/project/Art
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。