让大模型自动变简洁：用简单题约束推理长度，无需额外惩罚
Shorter but not Worse: Frugal Reasoning via Easy Samples as Length Regularizers in Math RLVR
- 用中等简单题作为隐式长度正则，控制模型输出长度
- 在不加长度惩罚下，解题长度减半，准确率不变
- 适合追求高效推理的数学推理场景
大型语言模型在逐步推理训练中常出现过度冗长问题，导致推理成本上升。标准的可验证奖励强化学习（RLVR）流程会过滤掉“简单”题目以提升效率，使模型主要在需要长推理链的难题上训练，从而导致输出长度分布偏高，模型误将‘思考更久’等同于‘思考更好’。本文提出保留并适度加权中等难度问题，可作为隐式的长度正则化手段。通过接触可解的短链任务，模型输出分布被有效约束，避免了无限制的冗长输出。实验表明，在不引入任何显式长度惩罚的情况下，模型实现了‘免费的涌现简洁性’：在使用具有16k上下文长度的Qwen3-4B-Thinking-2507模型进行测试时，保持基线pass@1 AIME25准确率的同时，解题平均长度接近减半。代码与数据已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) trained for step-by-step reasoning often become excessively verbose, raising inference cost. Standard Reinforcement Learning with Verifiable Rewards (RLVR) pipelines filter out ``easy'' problems for training efficiency, leaving the model to train primarily on harder problems that require longer reasoning chains. This skews the output length distribution upward, resulting in a \textbf{model that conflates ``thinking longer'' with ``thinking better''}. In this work, we show that retaining and modestly up-weighting moderately easy problems acts as an implicit length regularizer. Exposing the model to solvable short-chain tasks constrains its output distribution and prevents runaway verbosity. The result is \textbf{\emph{emergent brevity for free}}: the model learns to solve harder problems without inflating the output length, \textbf{ despite the absence of any explicit length penalization}. RLVR experiments using this approach on \textit{Qwen3-4B-Thinking-2507} (with a 16k token limit) achieve baseline pass@1 AIME25 accuracy while generating solutions that are, on average, nearly twice as short. The code is available at \href{https://github.com/MBZUAI-Paris/Frugal-AI}{GitHub}, with datasets and models on \href{https://huggingface.co/collections/MBZUAI-Paris/k2-think-mini-68dcfa8b114686a4bd3dc2bc}{Hugging Face}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。