通过判断推理片段是否接近正确答案，有效减少大模型的过度思考。
Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking
- 用中间答案与真值对比，低成本判断反思是否有效
- 在数学竞赛题上准确率提升至59.45%，减少无效思考
- 适合需要精准推理且怕冗余输出的场景
推理型语言模型常出现过度思考现象：生成大量无意义的自我质疑、放弃或自相矛盾的语句，消耗token却无助于提升答案质量。我们发现，这种行为并非仅由输出长度导致；即使控制长度，错误推理路径的无产反思率仍高于正确路径。解决此问题需识别反思何时有益、何时有害，但逐步标注成本高昂。我们观察到，推理过程中的中间答案承诺可作为廉价代理：通过将每个候选最终答案与真实答案比较，即可判断后续反思是否有效，无需额外标注。基于此，提出DASH（Drift Aware advantage SHaping）方法，根据每个推理段落是否趋向正确来分配段级奖励。在竞赛级数学基准测试中，DASH在过度思考普遍的场景下达到最高准确率（平均准确率59.45%），优于Dr.GRPO（58.1%）和GRPO（56.95%），同时显著减少过度思考，实现更有效的自我修正。
原文摘要 · Abstract (English)
Reasoning language models frequently overthink: generating extended chains of behaviors such as hedging, approach abandonment, and self contradiction that consume tokens without improving answers. We show that these behaviors are not merely a consequence of length; even when controlling for response length, incorrect traces exhibit higher rates of unproductive self-reflection than correct ones. Addressing this requires identifying where self-reflection helps vs hurts, but obtaining these step-level annotations is costly. We observe that intermediate answer commitments within reasoning traces can provide a cheap proxy: by comparing each final answer candidate in the trace to the ground truth, we can determine whether subsequent reflection is productive without any additional supervision. Building on this insight, we propose DASH (Drift Aware advantage SHaping), which assigns segment-level credit based on whether each reasoning segment leads toward or away from correctness. On competition-level math benchmarks, DASH achieves the highest accuracy where overthinking is prevalent (Average Accuracy: 59.45% vs. 58.1% Dr.GRPO vs. 56.95% GRPO) while reducing overthinking behaviors and achieving more productive self-correction than baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。