arXiv:2602.12113cs.AIcs.CL2026-02中稿 · ICLR被引 1

让小模型少废话,用动态惩罚机制提升推理效率与准确率。

Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty

  • 通过自适应反射惩罚和长度协调惩罚,动态控制推理过程
  • 1.5B模型响应长度减少53.1%,准确率提升5.8%;7B模型长度减35.0%,准确率升2.7%
  • 适合追求高效推理的轻量级模型部署,尤其数学推理任务

大型推理模型在复杂任务中表现优异,但常因重复提问、循环思考等冗余反思生成过长思维链,导致高消耗、高延迟且不提升准确率,尤其在小型模型中更明显。我们发现问题复杂度越高,冗余反思越严重,反而降低准确率并增加开销。为此提出ARLCP框架,通过强化学习动态平衡推理效率与准确性:一是自适应反射惩罚,抑制非必要反思步骤;二是按问题复杂度校准的长度惩罚,引导生成更简洁有效的推理路径。在五个数学推理基准上使用DeepSeek-R1-Distill-Qwen-1.5B和7B模型测试,结果表明:1.5B模型平均响应长度下降53.1%,准确率提升5.8%;7B模型长度减少35.0%,准确率提高2.7%。代码已开源。

原文摘要 · Abstract (English)

Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex reasoning tasks by employing test-time scaling. However, they often generate over-long chains-of-thought that, driven by substantial reflections such as repetitive self-questioning and circular reasoning, lead to high token consumption, substantial computational overhead, and increased latency without improving accuracy, particularly in smaller models. Our observation reveals that increasing problem complexity induces more excessive and unnecessary reflection, which in turn reduces accuracy and increases token overhead. To address this challenge, we propose Adaptive Reflection and Length Coordinated Penalty (ARLCP), a novel reinforcement learning framework designed to dynamically balance reasoning efficiency and solution accuracy. ARLCP introduces two key innovations: (1) a reflection penalty that adaptively curtails unnecessary reflective steps while preserving essential reasoning, and (2) a length penalty calibrated to the estimated complexity of the problem. By coordinating these penalties, ARLCP encourages the model to generate more concise and effective reasoning paths. We evaluate our method on five mathematical reasoning benchmarks using DeepSeek-R1-Distill-Qwen-1.5B and DeepSeek-R1-Distill-Qwen-7B models. Experimental results show that ARLCP achieves a superior efficiency-accuracy trade-off compared to existing approaches. For the 1.5B model, it reduces the average response length by 53.1% while simultaneously improving accuracy by 5.8%. For the 7B model, it achieves a 35.0% reduction in length with a 2.7% accuracy gain. The code is released at https://github.com/ZeweiYu1/ARLCP .

推理优化轻量化模型强化学习数学推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。