通过重采样让模型自动生成更短的推理链条，不改奖励函数也能提速。
QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization
- 用分象限重采样法，优先保留短正确、长错误回复
- 在1.5B到32B模型上缩短响应长度30%~70%
- 无需调参或改奖励函数，适合部署优化
近期大型推理模型在强化学习过程中常生成过长的思维链，导致推理延迟高、部署成本大。现有长度控制方法多依赖显式的长度惩罚或额外控制模块，需精细调参且可能损害推理质量。本文提出一种基于重采样的长度感知策略优化方法——QLPO，它是GRPO的简化变体，通过隐式方式实现长度控制而不修改奖励函数。QLPO先生成大量候选回复，再按经验正确/错误比例重采样，同时偏好短的正确回复和长的错误回复，从而重构训练分布，隐式鼓励生成更短输出。在1.5B至32B参数规模的基座模型与强推理模型上，QLPO一致提升了准确率与长度的权衡表现，响应长度减少30%至70%，同时保持推理性能。结果表明，结构化重采样是一种高效且鲁棒的推理优化路径。
原文摘要 · Abstract (English)
Recent large reasoning models often develop long chain-of-thought responses during reinforcement learning (RL), resulting in high inference latency and deployment cost. Existing methods for response length control typically rely on explicit length penalties or additional control modules, which require careful tuning and may compromise reasoning quality. We propose Quadrant-weighted Sampling for Length-aware Policy Optimization (QLPO), a simple resampling-based variant of GRPO that introduces implicit length control without modifying the reward function. QLPO first over-generates candidate responses and then resamples the training group by preserving the empirical correct/incorrect ratio while favoring short correct responses and long incorrect responses. This reshapes the training distribution and implicitly encourages shorter model outputs. Across models ranging from 1.5B to 32B parameters, including both base models and strong reasoning models, QLPO consistently improves the accuracy-length trade-off. It reduces response length by 30% to 70% while preserving reasoning performance. These results suggest that structured resampling provides an effective and robust approach to efficient reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。