让大模型根据问题难易自动调节思考深度,又快又准。
AdapThink: Adaptive Thinking Preferences for Reasoning Language Model
- 用动态奖励机制调节反思性词语使用,不固定思考长度。
- 在多个数学数据集上提升推理效率30%以上,准确率不变。
- 适合追求推理速度与效率的AI应用开发者使用。
基于强化学习的后训练显著提升了语言模型的复杂推理能力,促进了深层次自我反思过程。然而,这种“慢思考”范式带来了推理效率的关键挑战:模型可能在简单问题上过度计算,或在复杂问题上过早开始推理。现有方法通常依赖静态长度预算或预设规则,缺乏对问题复杂度和模型能力变化的适应性。为此,我们提出AdapThink,一种自适应后训练框架,旨在提升推理效率的同时保持模型性能。具体包含两项核心机制:1)群体相对奖励函数,利用模型置信度和响应特征动态调整反思相关过渡词的偏好,无需固定长度偏好;2)多样性感知采样机制,通过熵引导得分平衡训练组解题准确率与推理多样性。在多个数学推理数据集上使用DeepSeek微调模型的实验表明,AdapThink能有效实现自适应推理模式,缓解效率瓶颈。
原文摘要 · Abstract (English)
Reinforcement Learning (RL)-based post-training has significantly advanced the complex reasoning capabilities of language models, fostering sophisticated self-reflection processes. However, this ``slow thinking'' paradigm presents a critical challenge to reasoning efficiency: models may expend excessive computation on simple questions and shift reasoning prematurely for complex ones. Previous mechanisms typically rely on static length budgets or predefined rules, lacking the adaptability for varying question complexities and models' evolving capabilities. To this end, we propose AdapThink, an adaptive post-training framework designed to induce more efficient thinking while maintaining the performance of reasoning language models. Specifically, AdapThink incorporates two key mechanisms: 1) A group-relative reward function that leverages model confidence and response's characteristic to dynamically adjust the preference of reflection-related transition words without resorting to a fixed length preference. 2) A diversity-aware sampling mechanism that balances the training group's solution accuracy with reasoning diversity via an entropy-guided score. Experiments on several mathematical reasoning datasets with DeepSeek-distilled models demonstrate AdapThink's advantages in enabling adaptive reasoning patterns and mitigating the inefficiencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。