让大模型推理更短更准,自动删冗余步骤
Constraint-Rectified Training for Efficient Chain-of-Thought
- 用约束优化方法交替减少长度和修正错误,避免过度推理
- 推理长度减少30%以上,准确率保持稳定不下降
- 可生成不同详细程度的解释,无需重新训练
链式思维(CoT)显著提升了大语言模型的推理能力,尤其结合强化学习后训练方法时。虽然更长的推理过程能提高答案质量并实现自校正,但也会带来高昂的推理开销,并常引入冗余步骤(过思考)。现有研究通过长度感知奖励或提示校准来平衡推理长度与准确性,但这些基于启发式的方法易导致准确率大幅下降,且对超参数敏感。为此,我们提出CRT(约束修正训练),一种基于参考引导的约束优化后训练框架,提供更稳定、可解释的高效推理方案。CRT在推理长度最小化与性能低于参考值时修正准确率之间交替进行,实现稳定有效的冗余推理剪枝。我们进一步设计两阶段训练:先发现最短可靠的推理模式,再在学习到的长度预算下优化准确率,防止冗长CoT重现。全面评估表明,该框架持续降低令牌使用量,同时维持高且可靠的准确率。进一步分析显示,CRT不仅缩短输出,还减少内部语言冗余,催生新评估指标。此外,基于CRT的训练自然产生一系列中间检查点,覆盖从简到繁的解释层次,实现无需重训的推理冗余精细控制。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs), especially when combined with reinforcement learning (RL) based post-training methods. While longer reasoning traces can improve answer quality and unlock abilities such as self-correction, they also incur high inference costs and often introduce redundant steps, known as overthinking. Recent research seeks to develop efficient reasoning strategies that balance reasoning length and accuracy, either through length-aware reward design or prompt-based calibration. However, these heuristic-based approaches may suffer from severe accuracy drop and be very sensitive to hyperparameters. To address these problems, we introduce CRT (Constraint-Rectified Training), a principled post-training framework based on reference-guarded constrained optimization, yielding a more stable and interpretable formulation for efficient reasoning. CRT alternates between minimizing reasoning length and rectifying accuracy only when performance falls below the reference, enabling stable and effective pruning of redundant reasoning. We further extend CRT with a two-stage training scheme that first discovers the shortest reliable reasoning patterns and then refines accuracy under a learnt length budget, preventing the re-emergence of verbose CoT. Our comprehensive evaluation shows that this framework consistently reduces token usage while maintaining answer quality at a robust and reliable level. Further analysis reveals that CRT improves reasoning efficiency not only by shortening responses but also by reducing internal language redundancy, leading to a new evaluation metric. Moreover, CRT-based training naturally yields a sequence of intermediate checkpoints that span a spectrum of explanation lengths while preserving correctness, enabling fine-grained control over reasoning verbosity without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。