让大模型快速思考，减少废话和冗余推理。
Towards Flash Thinking via Decoupled Advantage Policy Optimization
- 拆分优势信号，精准删减无用推理步骤。
- 响应长度减少39%，准确率反而更高。
- 适合追求高效推理的部署场景。
近期的大规模推理模型（LRMs）通过监督微调（SFT）和强化学习（RL）在解决复杂问题上取得了显著进展。尽管现有强化学习算法显著提升了模型准确率，但仍存在响应过长、过度思考等问题，导致推理延迟和计算开销增加，尤其对简单任务尤为明显。为此，我们提出一种新型强化学习框架DEPO，以减少模型的低效推理。该方法包含三个核心组件：(1) 创新的优势解耦算法，引导模型剔除低效标记；(2) 基于难度感知的长度惩罚机制，降低整体响应长度；(3) 优势裁剪方法，防止策略优化中的偏差。实验中，将DEPO应用于DeepSeek-Distill-Qwen-7B和DeepSeek-Distill-Qwen-1.5B作为基线模型，结果显示序列长度显著减少39%，同时减少了无效推理路径，且整体准确率优于基线模型。
原文摘要 · Abstract (English)
Recent Large Reasoning Models (LRMs) have achieved remarkable performance in solving complex problems via supervised fine-tuning (SFT) and reinforcement learning (RL). Although existing RL algorithms significantly enhance model accuracy, they still suffer from excessively lengthy responses and overthinking issues, resulting in increased inference latency and computational consumption, especially for simple tasks that require minimal reasoning. To address this, we propose a novel RL framework, DEPO, to reduce inefficient reasoning for models. Our method mainly consists of three core components: (1) an innovative advantage decoupled algorithm to guide model reduction of inefficient tokens; (2) a difficulty-aware length penalty to lower the overall length of model responses; (3) an advantage clipping method to prevent bias in policy optimization. In our experiments, applied to DeepSeek-Distill-Qwen-7B and DeepSeek-Distill-Qwen-1.5B as base models, DEPO achieves a significant reduction in sequence length by 39% and reduces excessive reasoning paths in inefficient tokens, while outperforming the base model in overall accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。