让大模型推理更高效,同时不牺牲准确性
ADaPT: Token-Level Decoupling for Efficient Large Reasoning Models

- 在词元层面分离效率与正确性信号,动态选择快慢推理模式
- 仅对控制令牌奖励效率,避免惩罚长而正确的推理路径
- 推理时可平滑调节效率与性能,单模型适配多种需求
大型推理模型依赖长链思维实现强性能,但统一应用此类推理会带来高计算开销。现有高效方法试图缩短或混合推理策略,却常损害推理能力。我们发现根本原因是效率激励与正确性优化在序列层面耦合,隐式惩罚了虽长但正确的推理轨迹。为此,提出自适应双过程思维(ADaPT),一种词元级双过程框架,在训练中显式解耦效率与正确性信号。ADaPT引入模式选择词元以控制快速与慢速推理,仅对该词元施加效率相关奖励,避免惩罚正确长推理,同时在适当情况下鼓励效率。此外,ADaPT可在推理时实现精确且连续的效率-性能权衡控制:通过调整模式选择词元的生成概率,单一训练模型可沿效率-性能帕累托前沿平滑移动。大量实验表明,ADaPT显著降低推理成本,同时在多个基准上保持强大推理性能。
原文摘要 · Abstract (English)
Large reasoning models rely on long chain-of-thought to achieve strong performance, but applying such reasoning uniformly incurs high computational cost. Existing efficiency-oriented methods attempt to shorten or mix reasoning strategies, yet often degrade reasoning capability. We identify the root cause as sequence-level coupling between efficiency incentives and correctness optimization, which implicitly penalizes long but correct reasoning trajectories. To address this issue, we propose Adaptive Dual-Process Thinking (ADaPT), a token-level dual-process framework that explicitly decouples efficiency and correctness signals during training. ADaPT introduces a mode-selection token to control fast and slow reasoning, applying efficiency-related rewards exclusively to this token to avoid penalizing correct long reasoning while encouraging efficiency when appropriate. Moreover, ADaPT enables precise and continuous control over the efficiency-performance trade-off at inference time: by adjusting the generation probability of the mode-selection token, a single trained model can smoothly move along the efficiency-performance Pareto frontier. Extensive experiments demonstrate that ADaPT significantly reduces inference cost while maintaining strong reasoning performance across multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。