o1类大模型常因思维跳跃导致解题不深,新方法有效提升数学难题准确率。
Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs
- 通过惩罚思维切换频率,引导模型深入探索每条推理路径。
- 在三个难题数据集上,准确率显著提升,错误答案的令牌效率降低37%。
- 无需微调模型,适合希望提升推理深度的开发者与研究者使用。
如OpenAI o1等大型语言模型通过扩展推理时计算量,在复杂推理任务中展现出类人深度思考能力。然而,我们发现一种称为‘欠思考’的现象:o1类模型频繁在不同推理思路间切换,未能充分探索有潜力的路径,导致推理深度不足,尤其在高难度数学问题上表现下降。为系统分析此问题,我们在三个挑战性测试集和两个代表性开源o1类模型上开展实验,发现频繁思维切换与错误答案密切相关。为此,我们提出一种新指标,通过衡量错误答案中的令牌效率来量化欠思考程度。针对该问题,我们设计了一种带思维切换惩罚(TIP)的解码策略,抑制过早切换,促进对每条推理路径的深入探索。实验表明,该方法在不需模型微调的情况下,显著提升了多个挑战性数据集上的准确率。研究成果有助于理解o1类模型的推理瓶颈,并提供可直接应用的优化方案。
原文摘要 · Abstract (English)
Large language models (LLMs) such as OpenAI's o1 have demonstrated remarkable abilities in complex reasoning tasks by scaling test-time compute and exhibiting human-like deep thinking. However, we identify a phenomenon we term underthinking, where o1-like LLMs frequently switch between different reasoning thoughts without sufficiently exploring promising paths to reach a correct solution. This behavior leads to inadequate depth of reasoning and decreased performance, particularly on challenging mathematical problems. To systematically analyze this issue, we conduct experiments on three challenging test sets and two representative open-source o1-like models, revealing that frequent thought switching correlates with incorrect responses. We introduce a novel metric to quantify underthinking by measuring token efficiency in incorrect answers. To address underthinking, we propose a decoding strategy with thought switching penalty TIP that discourages premature transitions between thoughts, encouraging deeper exploration of each reasoning path. Experimental results demonstrate that our approach improves accuracy across challenging datasets without requiring model fine-tuning. Our findings contribute to understanding reasoning inefficiencies in o1-like LLMs and offer a practical solution to enhance their problem-solving capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。