动态调整惩罚系数，让大模型推理更短更准。
Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model
- 用拉格朗日对偶法动态调节长度惩罚，随生成长度自动增减。
- 在多个任务上平均缩短60%推理长度，性能基本不变。
- 适合需要高效推理的场景，如部署在资源受限设备上。
现有方法多依赖固定长度惩罚，难以适应大模型推理能力的变化，导致准确率与简洁性难以兼顾。为此，我们提出Leash（自适应长度惩罚与奖励塑造）框架，将长度控制建模为约束优化问题，采用拉格朗日对偶方法动态调整惩罚系数：生成过长时加强惩罚，过短时降低惩罚。该机制引导模型生成更简洁的推理过程而不牺牲任务表现。在Deepseek-R1-Distill-Qwen-1.5B和Qwen3-4B-Thinking-2507上的实验表明，Leash在多种任务（包括数学推理、编程、指令遵循等）中平均将推理长度减少60%，同时保持竞争力性能。本工作为实现可控且高效的大型语言模型提供了一种实用有效的新范式。
原文摘要 · Abstract (English)
Existing approaches typically rely on fixed length penalties, but such penalties are hard to tune and fail to adapt to the evolving reasoning abilities of LLMs, leading to suboptimal trade-offs between accuracy and conciseness. To address this challenge, we propose Leash (adaptive LEngth penAlty and reward SHaping), a reinforcement learning framework for efficient reasoning in LLMs. We formulate length control as a constrained optimization problem and employ a Lagrangian primal-dual method to dynamically adjust the penalty coefficient. When generations exceed the target length, the penalty is intensified; when they are shorter, it is relaxed. This adaptive mechanism guides models toward producing concise reasoning without sacrificing task performance. Experiments on Deepseek-R1-Distill-Qwen-1.5B and Qwen3-4B-Thinking-2507 show that Leash reduces the average reasoning length by 60% across diverse tasks - including in-distribution mathematical reasoning and out-of-distribution domains such as coding and instruction following - while maintaining competitive performance. Our work thus presents a practical and effective paradigm for developing controllable and efficient LLMs that balance reasoning capabilities with computational budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。