让大模型生成更短回答，同时保持准确率。
LACONIC: Length-Aware Constrained Reinforcement Learning for LLM
- 用动态成本控制生成长度，避免硬性规则
- 数学推理任务输出减少50%以上，准确率不变
- 无需修改推理流程，适合实际部署
强化学习提升了大语言模型的能力，但常导致响应过长，增加推理延迟和计算开销。现有长度控制方法依赖固定启发式奖励设计，易与任务目标偏离且调参脆弱。本文提出LACONIC，一种在训练中强制目标词元预算的强化学习方法。通过将任务奖励与长度成本相结合的增强目标更新策略，动态调整成本权重，在保证任务性能的同时实现鲁棒长度控制。该方法具备理论保障。在数学推理模型与数据集上，LACONIC在保持或提升pass@1准确率的同时，输出长度减少超过50%；在通用知识与多语言基准测试中，以44%更少的词元维持跨域性能。LACONIC可无缝集成至标准强化学习微调流程，无推理阶段改动，部署开销极低。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has enhanced the capabilities of large language models (LLMs) through reward-driven training. Nevertheless, this process can introduce excessively long responses, inflating inference latency and computational overhead. Prior length-control approaches typically rely on fixed heuristic reward shaping, which can misalign with the task objective and require brittle tuning. In this work, we propose LACONIC, a reinforcement learning method that enforces a target token budget during training. Specifically, we update policy models using an augmented objective that combines the task reward with a length-based cost. To balance brevity and task performance, the cost scale is adaptively adjusted throughout training. This yields robust length control while preserving task reward. We provide a theoretical guarantee that support the method. Across mathematical reasoning models and datasets, LACONIC preserves or improves pass@1 while reducing output length by over 50%. It maintains out-of-domain performance on general knowledge and multilingual benchmarks with 44% fewer tokens. Moreover, LACONIC integrates into standard RL-tuning with no inference changes and minimal deployment overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。