让大模型自动判断该多想还是少想，省下一半算力还不丢分。
Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning
- 用强化学习动态调整每道题的生成长度惩罚，难的多给算力，易的少耗资源。
- 在DeepScaleR-1.5B上实验，平均节省50%生成token，准确率基本不变。
- 适合追求高效推理的开发者，尤其在资源受限场景下表现更优。
大型推理模型（LRMs）通过增加推理时的生成标记数，在复杂推理任务上取得更高性能，但这种冗长性常在简单问题上浪费计算资源。现有方法如基于短推理路径的监督微调、用户设定的预算限制，或使用统一惩罚的强化学习，均需数据整理、人工配置，或对所有问题一视同仁。本文提出自适应长度惩罚（ALP），一种基于强化学习的目标，根据每个提示的实时解题率动态调节生成长度。训练期间，ALP通过多次推演监测每个提示的在线解题率，并施加可微惩罚，其强度与解题率成反比：解题率高的（即简单问题）会因额外生成而承受更高代价，而难题则不受影响。使用ALP训练后的DeepScaleR-1.5B模型，在不显著降低性能的前提下，将平均标记使用量减少50%。相比固定预算和均匀惩罚基线，ALP更智能地重新分配计算资源，对简单问题压缩算力，将节省的标记重新分配给困难问题，从而在高成本难题上实现更高准确率。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) achieve higher performance on challenging reasoning tasks by generating more tokens at inference time, but this verbosity often wastes computation on easy problems. Existing solutions, including supervised finetuning on shorter traces, user-controlled budgets, or RL with uniform penalties, either require data curation, manual configuration, or treat all problems alike regardless of difficulty. We introduce Adaptive Length Penalty (ALP), a reinforcement learning objective tailoring generation length to per-prompt solve rate. During training, ALP monitors each prompt's online solve rate through multiple rollouts and adds a differentiable penalty whose magnitude scales inversely with that rate, so confident (easy) prompts incur a high cost for extra tokens while hard prompts remain unhindered. Posttraining DeepScaleR-1.5B with ALP cuts average token usage by 50\% without significantly dropping performance. Relative to fixed-budget and uniform penalty baselines, ALP redistributes its reduced budget more intelligently by cutting compute on easy prompts and reallocating saved tokens to difficult ones, delivering higher accuracy on the hardest problems with higher cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。