让AI推理更简洁:动态调整惩罚机制,既省字数又不丢分。
Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning
- 在正确答案中动态计算长度优势,避免错误答案拉低基准线。
- 根据题目难易程度自动调节惩罚强度,简单题省60%字数,难题省20%以上。
- 适合追求高效精准推理的开发者,尤其适用于数学证明类任务。
强化学习结合可验证奖励(RLVR)能激发强大的多步推理能力,但常导致输出过于冗长。组内相对优化中的简单长度惩罚会严重损害准确率。我们归因于两个结构性问题:(i) 长度基准稀释——错误回答(零长度奖励)拉低组内基准,过度惩罚正确解;(ii) 难度-惩罚不匹配——静态惩罚无法适应题目难度,抑制难题必要推理,却对简单题冗余无用。为此提出动态解耦条件优势(DDCA),将效率优化与正确性解耦:在正确回答簇内条件化计算长度优势,消除基准稀释;并以组通过率作为难度代理,动态调节惩罚强度。在GSM8K、MATH500、AMC23和AIME25上的实验表明,相比自适应基线,DDCA始终提升效率-准确率权衡,在简单任务(如GSM8K)上减少约60%生成词元,在困难任务(如AIME25)上减少超20%,同时保持或提升准确率。代码已公开于https://github.com/alphadl/DDCA。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) can elicit strong multi-step reasoning, yet it often encourages overly verbose traces. Moreover, naive length penalties in group-relative optimization can severely hurt accuracy. We attribute this failure to two structural issues: (i) Dilution of Length Baseline, where incorrect responses (with zero length reward) depress the group baseline and over-penalize correct solutions; and (ii) Difficulty-Penalty Mismatch, where a static penalty cannot adapt to problem difficulty, suppressing necessary reasoning on hard instances while leaving redundancy on easy ones. We propose Dynamic Decoupled Conditional Advantage (DDCA) to decouple efficiency optimization from correctness. DDCA computes length advantages conditionally within the correct-response cluster to eliminate baseline dilution, and dynamically scales the penalty strength using the group pass rate as a proxy for difficulty. Experiments on GSM8K, MATH500, AMC23, and AIME25 show that DDCA consistently improves the efficiency--accuracy trade-off relative to adaptive baselines, reducing generated tokens by approximately 60% on simpler tasks (e.g., GSM8K) versus over 20% on harder benchmarks (e.g., AIME25), thereby maintaining or improving accuracy. Code is available at https://github.com/alphadl/DDCA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。