让大模型自动判断问题难易,该简则简该繁则繁,兼顾准确与效率
DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains
- 根据问题难易动态调整推理长度,简单题缩短路径,难题拓展思考
- 在数学基准上同时提升准确率与节省40%以上推理token
- 适合追求高效精准推理的AI系统开发者或研究者使用
大型推理模型虽表现优异,却存在对简单问题过度思考、对复杂问题思考不足的认知低效问题。现有基于监督微调或强化学习的方法虽能提升效率,常以牺牲准确率为代价。本文提出DeepCompress框架,通过自适应长度奖励机制,在实时判断问题为‘简单’或‘困难’的基础上,对简单问题鼓励短路径推理,对困难问题支持长路径探索。该双奖励策略使模型能自主调节思维链长度,对已掌握问题压缩推理,对挑战性问题延长思考。在多个高难度数学基准上的实验表明,DeepCompress持续优于基线方法,在保持更高准确率的同时,显著提升令牌效率,平均节省超过40%的推理计算量。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have demonstrated impressive capabilities but suffer from cognitive inefficiencies like "overthinking" simple problems and "underthinking" complex ones. While existing methods that use supervised fine-tuning (SFT) or reinforcement learning (RL) with token-length rewards can improve efficiency, they often do so at the cost of accuracy. This paper introduces DeepCompress, a novel framework that simultaneously enhances both the accuracy and efficiency of LRMs. We challenge the prevailing approach of consistently favoring shorter reasoning paths, showing that longer responses can contain a broader range of correct solutions for difficult problems. DeepCompress employs an adaptive length reward mechanism that dynamically classifies problems as "Simple" or "Hard" in real-time based on the model's evolving capability. It encourages shorter, more efficient reasoning for "Simple" problems while promoting longer, more exploratory thought chains for "Hard" problems. This dual-reward strategy enables the model to autonomously adjust its Chain-of-Thought (CoT) length, compressing reasoning for well-mastered problems and extending it for those it finds challenging. Experimental results on challenging mathematical benchmarks show that DeepCompress consistently outperforms baseline methods, achieving superior accuracy while significantly improving token efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。