测试时多思考未必更好,过度推理反而降低准确率。
Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models
- 提出并验证了测试时扩展推理会先升后降的非单调现象
- 发现过度推理增加输出方差,导致准确率下降20%以上
- 用并行推理投票替代串行重思,更有效利用推理资源
近期推理模型的测试时扩展(如OpenAI o1、DeepSeek R1)流行一种观点:通过添加‘等待’或‘让我重新思考’等提示延长推理过程可提升性能。我们通过跨模型与基准的实证研究发现,额外推理初期表现上升,随后下降,源于‘过度思考’。基于简单概率模型分析表明,额外推理会增加输出方差,制造出性能改善的假象,实际削弱精度。这种增益并非真实推理能力提升,而是模型不确定性与评估指标关联所致。因此,单纯延长推理路径并非高效利用推理预算的方式。为此,我们提出受Best-of-N启发的并行推理方法:在相同预算内生成多个独立推理路径,通过多数投票选择最一致结果,相比串行重思最高提升20%准确率,提供了一种简单有效的测试时扩展新范式。
原文摘要 · Abstract (English)
Recent trends in test-time scaling for reasoning models (e.g., OpenAI o1, DeepSeek R1) have led to a popular belief that extending thinking traces using prompts like "Wait" or "Let me rethink" can improve performance. This raises a natural question: Does thinking more at test-time truly lead to better reasoning? To answer this question, we perform a detailed empirical study across models and benchmarks, which reveals a consistent pattern of initial performance improvements from additional thinking followed by a decline, due to "overthinking". To understand this non-monotonic trend, we consider a simple probabilistic model, which reveals that additional thinking increases output variance-creating an illusion of improved reasoning while ultimately undermining precision. Thus, observed gains from "more thinking" are not true indicators of improved reasoning, but artifacts stemming from the connection between model uncertainty and evaluation metric. This suggests that test-time scaling through extended thinking is not an effective way to utilize the inference thinking budget. Recognizing these limitations, we introduce an alternative test-time scaling approach, parallel thinking, inspired by Best-of-N sampling. Our method generates multiple independent reasoning paths within the same inference budget and selects the most consistent response via majority vote, achieving up to 20% higher accuracy compared to extended thinking. This provides a simple yet effective mechanism for test-time scaling of reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。