让AI根据任务难易自动调节思考深度,兼顾速度与准确率
e1: Learning Adaptive Control of Reasoning Effort
- 用强化学习训练模型按比例分配思考资源,无需预设具体长度
- 在1.5B到32B参数模型上,思考链缩短2-3倍仍保持或提升性能
- 用户可动态调整推理成本,适合对响应速度敏感的应用
增加AI模型的思考预算能显著提升准确性,但并非所有问题都需同等推理量。用户常需在输出质量、延迟和成本间权衡。现有方法要求用户指定具体令牌数,但需预先知晓问题难度,难以灵活调整。为此,我们提出自适应努力控制,一种基于强化学习的自适应方法,使模型根据当前思维链平均长度,按用户设定的比例使用令牌。该方法避免了数据集和阶段相关的调优,相比标准方法产生更优的成本-准确率曲线。用户可在推理时通过连续参数动态调节成本-准确率平衡。实验发现,模型自动按任务难度分配资源;在1.5B至32B参数规模下,思考链长度减少2-3倍,同时性能不降或优于基础模型。
原文摘要 · Abstract (English)
Increasing the thinking budget of AI models can significantly improve accuracy, but not all questions warrant the same amount of reasoning. Users may prefer to allocate different amounts of reasoning effort depending on how they value output quality versus latency and cost. To leverage this tradeoff effectively, users need fine-grained control over the amount of thinking used for a particular query, but few approaches enable such control. Existing methods require users to specify the absolute number of desired tokens, but this requires knowing the difficulty of the problem beforehand to appropriately set the token budget for a query. To address these issues, we propose Adaptive Effort Control, a self-adaptive reinforcement learning method that trains models to use a user-specified fraction of tokens relative to the current average chain-of-thought length for each query. This approach eliminates dataset- and phase-specific tuning while producing better cost-accuracy tradeoff curves compared to standard methods. Users can dynamically adjust the cost-accuracy trade-off through a continuous effort parameter specified at inference time. We observe that the model automatically learns to allocate resources proportionally to the task difficulty and, across model scales ranging from 1.5B to 32B parameters, our approach enables a 2-3x reduction in chain-of-thought length while maintaining or improving performance relative to the base model used for RL training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。