对比串行与并行搜索,发现回溯并非总能提升大模型推理能力。
To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoning
- 用串行链式思考与并行采样对比推理效率
- 在数独任务中串行优于并行,在计数挑战中则相反
- 训练数据和显式监督会抑制模型隐性推理能力
大语言模型的推理能力近年显著提升,尤其依赖搜索与回溯技术。回溯通过生成长链式思维(CoT)实现测试时计算的线性扩展,但并行采样结合最佳N选一也是可选方案。尽管串行搜索被广泛采用,其在固定计算预算下相比并行采样的优势仍不明确。本文在两个高难度推理任务——计数挑战(CountDown)与数独(Sudoku)上系统比较两种策略。结果出人意料:串行搜索在计数挑战中表现更差,但在数独中表现更优,表明回溯并非普遍有益。我们识别出两个导致回溯性能下降的因素:(1) 在固定搜索路径上训练会使模型固化于次优策略;(2) 显式CoT监督会抑制隐性(非显式表达)推理。进一步在强化学习(RL)场景分析显示,具备回溯能力的模型经RL微调后收益显著,而无回溯能力者仅获有限且混杂的提升。这些发现挑战了回溯普遍增强推理的假设,揭示任务结构、训练数据、模型规模与学习范式间的复杂交互关系。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have significantly improved their reasoning abilities, particularly through techniques involving search and backtracking. Backtracking naturally scales test-time compute by enabling sequential, linearized exploration via long chain-of-thought (CoT) generation. However, this is not the only strategy for scaling test time-compute: parallel sampling with best-of-N selection provides an alternative that generates diverse solutions simultaneously. Despite the growing adoption of sequential search, its advantages over parallel sampling-especially under a fixed compute budget-remain poorly understood. In this paper, we systematically compare these two approaches on two challenging reasoning tasks: CountDown and Sudoku. Surprisingly, we find that sequential search underperforms parallel sampling on CountDown but outperforms it on Sudoku, suggesting that backtracking is not universally beneficial. We identify two factors that can cause backtracking to degrade performance: (1) training on fixed search traces can lock models intro suboptimal strategies, and (2) explicit CoT supervision can discourage implicit (non verbalized) reasoning. Extending our analysis to reinforcement learning (RL), we show that models with backtracking capabilities benefit significantly from RL fine-tuning, while models without backtracking see limited, mixed gains. Together, these findings challenge the assumption that backtracking universally enhances LLM reasoning, instead revealing a complex interaction between task structure, training data, model scale, and learning paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。