通过步骤级优势选择，让大模型推理更高效稳定。
Stabilizing Efficient Reasoning with Step-Level Advantage Selection
- 在每步推理中动态判断信心度，决定是否保留
- 推理长度减少16.3%，准确率提升0.86点
- 适合追求高效可靠推理的落地应用
大型语言模型在推理时通过生成长而冗长的推理链来实现优异表现。尽管近期高效推理方法通过长度奖励或剪枝减少计算开销，但许多方法在远短于基础训练上下文窗口的条件下进行后训练，这一因素尚未被系统分析。我们首先发现，仅使用标准GRPO且无长度感知目标进行短上下文后训练，已能显著压缩推理过程，但伴随训练动态不稳和准确率下降。为此，我们提出步骤级优势选择（SAS），在推理步骤层面分配优势：对正确路径中低置信度步骤赋予零优势，对验证失败路径中高置信度步骤也赋予零优势，因失败多由截断或验证问题导致而非推理错误。在多个数学与通用推理基准上，SAS相比最强长度感知基线平均提升Pass@1准确率0.86点，同时将平均推理长度降低16.3%，实现更优的准确性-效率权衡。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve strong reasoning performance by allocating substantial computation at inference time, often generating long and verbose reasoning traces. While recent work on efficient reasoning reduces this overhead through length-based rewards or pruning, many approaches are post-trained under a much shorter context window than base-model training, a factor whose effect has not been systematically isolated. We first show that short-context post-training alone, using standard GRPO without any length-aware objective, already induces substantial reasoning compression-but at the cost of increasingly unstable training dynamics and accuracy degradation. To address this, we propose Step-level Advantage Selection (SAS), which operates at the reasoning-step level and assigns a zero advantage to low-confidence steps in correct rollouts and to high-confidence steps in verifier-failed rollouts, where failures often arise from truncation or verifier issues rather than incorrect reasoning. Across diverse mathematical and general reasoning benchmarks, SAS improves average Pass@1 accuracy by 0.86 points over the strongest length-aware baseline while reducing average reasoning length by 16.3%, yielding a better accuracy-efficiency trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。