arXiv:2505.20258cs.CL2025-05NeurIPS被引 29

让大模型根据任务难易自动选推理方式,省 token 又快

ARM: Adaptive Reasoning Model

  • 自适应选择直接回答、短思维链、代码或长思维链四种推理格式
  • 平均节省30%令牌,最多达70%,性能媲美纯长思维链
  • 支持用户指定格式或多数表决,兼顾效率与准确率

大型推理模型在复杂任务上表现强劲,却无法根据任务难度调整推理令牌使用,常导致‘过度思考’问题——产生冗余推理。尽管可通过人工控制令牌预算缓解,但违背了实现完全自主AI的目标。本文提出自适应推理模型(ARM),能根据任务动态选择合适推理格式:包括三种高效格式(直接回答、短思维链、代码)和一种更复杂的长思维链。为训练ARM,引入Ada-GRPO,改进传统组相对策略优化(GRPO),解决格式坍缩问题。实验显示,ARM平均减少30%令牌消耗,最高达70%,且性能与仅用长思维链的模型相当。同时,推理效率提升,训练速度加快2倍。除默认自适应模式外,还支持指令引导模式(用户通过特殊标记指定格式)和共识引导模式(三类高效格式输出不一致时启用长思维链),平衡效率与准确性。

原文摘要 · Abstract (English)

While large reasoning models demonstrate strong performance on complex tasks, they lack the ability to adjust reasoning token usage based on task difficulty. This often leads to the "overthinking" problem -- excessive and unnecessary reasoning -- which, although potentially mitigated by human intervention to control the token budget, still fundamentally contradicts the goal of achieving fully autonomous AI. In this work, we propose Adaptive Reasoning Model (ARM), a reasoning model capable of adaptively selecting appropriate reasoning formats based on the task at hand. These formats include three efficient ones -- Direct Answer, Short CoT, and Code -- as well as a more elaborate format, Long CoT. To train ARM, we introduce Ada-GRPO, an adaptation of Group Relative Policy Optimization (GRPO), which addresses the format collapse issue in traditional GRPO. Ada-GRPO enables ARM to achieve high token efficiency, reducing tokens by an average of 30%, and up to 70%, while maintaining performance comparable to the model that relies solely on Long CoT. Furthermore, not only does it improve inference efficiency through reduced token generation, but it also brings a 2x speedup in training. In addition to the default Adaptive Mode, ARM supports two additional reasoning modes: 1) Instruction-Guided Mode, which allows users to explicitly specify the reasoning format via special tokens -- ideal when the appropriate format is known for a batch of tasks. 2) Consensus-Guided Mode, which aggregates the outputs of the three efficient formats and resorts to Long CoT in case of disagreement, prioritizing performance with higher token usage.

推理优化自适应令牌效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。