用强化学习让大模型按需思考，控制推理长度
L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
- 通过强化学习优化推理长度与准确率的平衡
- 实现可调长度推理，性能优于现有最先进方法
- 发现短推理模型也能高效思考，适合资源受限场景
推理型语言模型在测试时通过生成更长的思维链可显著提升性能，但其推理长度不可控，无法根据需求分配计算资源。本文提出长度可控策略优化（LCPO），一种简单的强化学习方法，同时优化准确率和用户指定的长度约束。基于此训练出的L1模型可在提示中指定长度，实现计算成本与精度的平滑权衡，在多种任务上表现优于当前最先进的S1方法。此外，我们意外发现使用LCPO训练的模型具备短思维链能力，由此衍生出短推理模型（SRMs），其推理模式与全长度模型相似，但思维链长度接近非推理模型。例如，1.5B参数的L1模型在等长推理下超越GPT-4o。LCPO实现了对推理长度的精确控制，支持细粒度的测试时计算资源配置。代码与模型已公开。
原文摘要 · Abstract (English)
Reasoning language models have shown an uncanny ability to improve performance at test-time by ``thinking longer''-that is, by generating longer chain-of-thought sequences and hence using more compute. However, the length of their chain-of-thought reasoning is not controllable, making it impossible to allocate test-time compute to achieve a desired level of performance. We introduce Length Controlled Policy Optimization (LCPO), a simple reinforcement learning method that optimizes for accuracy and adherence to user-specified length constraints. We use LCPO to train L1, a reasoning language model that produces outputs satisfying a length constraint given in its prompt. L1's length control allows for smoothly trading off computational cost and accuracy on a wide range of tasks, and outperforms the state-of-the-art S1 method for length control. Furthermore, we uncover an unexpected short chain-of-thought capability in models trained with LCPO. Specifically, using LCPO we derive Short Reasoning Models (SRMs), that exhibit similar reasoning patterns as full-length reasoning models, but can generate CoT lengths comparable to non-reasoning models. They demonstrate significant performance gains, for instance, our 1.5B L1 model surpasses GPT-4o at equal reasoning lengths. Overall, LCPO enables precise control over reasoning length, allowing for fine-grained allocation of test-time compute and accuracy. We release code and models at https://www.cmu-l3.github.io/l1
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。