发现模型反思多数不改答案,提出早停策略提升推理效率
First Try Matters: Revisiting the Role of Reflection in Reasoning Models
- 分析八种模型在五数据集上的反思行为,发现多为确认性反思
- 训练时增加反思步数只提高初答正确率,难纠正错误答案
- 提出动态截断策略,减少24.5%推理token,准确率仅降2.9%
大语言模型在推理能力上取得显著进步,常归因于生成更长思维链和反思能力。然而,反思对性能提升的实际贡献仍不明确。本文系统分析了八种推理模型在五个数学数据集上的推理过程,重点关注模型已生成答案后继续反思的行为。结果表明,反思行为以确认为主,极少改变初始答案,这一模式在不同模型和数据集间保持一致。为理解反思在训练中的作用,我们构建了包含不同反思步数的监督微调(SFT)数据集。发现训练时使用更多反思步数主要提升了初答正确率,而非通过反思修正错误答案的能力。基于此,我们提出一种问题感知的早停方法,在生成若干合理候选答案后停止推理,提升推理阶段的词元效率。进一步提出在生成过程中动态截断反思步骤,使五个数学数据集上的推理词元减少24.5%,准确率仅下降2.9%。
原文摘要 · Abstract (English)
Large language models have recently demonstrated significant gains in reasoning ability, often attributed to their capacity to generate longer chains of thought and engage in reflective reasoning. However, the contribution of reflections to performance improvement remains unclear. In this paper, we systematically analyze the rollouts of eight reasoning models on five mathematical datasets. We focus on reflective behaviours where the model has already produced an answer but continues reflecting before finalizing its output. Our analysis reveals that reflections are predominantly confirmatory and rarely alter the model's initial answer, a pattern consistent across models and datasets. To understand the role of reflections in training, we construct supervised fine-tuning (SFT) datasets with varying amounts of reflection steps. We observe that training models on rollouts with more reflection steps primarily enhances first-answer correctness rather than the ability to correct initially wrong answers through reflections. This motivates us to propose a question-aware early-stopping method that enhances inference-time token efficiency by stopping the reasoning process once a few plausible candidate answers are generated, thereby reducing unnecessary reflection steps. Motivated by this, we further propose to dynamically truncate the reflections after a candidate answer has appeared during generation, which reduces reasoning tokens by 24.5% across five mathematical datasets, within a 2.9% drop in accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。