统一树搜索与奖励设计，让大模型推理更高效自进化
Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey
- 将搜索算法拆解为机制、奖励、转移三部分，建立统一框架
- 区分临时引导与持久学习，解决奖励作用模糊问题
- 适合研究大模型推理与自我优化的学者参考
推理型树搜索是现代大语言模型研究的核心，推动了从盲目扩展向算法效率的转变。该范式统一了两个关键方向：测试时扩展（TTS），即按需计算解决难题；以及自提升，利用搜索生成的数据持续优化模型参数。然而，该领域碎片化严重，缺乏统一形式化，尤其在奖励信号角色上存在模糊——它是临时启发式，还是持久学习目标？本文通过引入统一框架，将搜索算法解构为三个核心组件：搜索机制、奖励表述和转移函数。我们建立了临时搜索引导（用于TTS）与持久参数化奖励建模（用于自提升）的形式区分。基于此，提出以组件为中心的分类体系，综合当前最先进方法，并规划未来研究路线图，以实现更系统化的自主、自提升智能体构建。
原文摘要 · Abstract (English)
Deliberative tree search is a cornerstone of modern Large Language Model (LLM) research, driving the pivot from brute-force scaling toward algorithmic efficiency. This single paradigm unifies two critical frontiers: \textbf{Test-Time Scaling (TTS)}, which deploys on-demand computation to solve hard problems, and \textbf{Self-Improvement}, which uses search-generated data to durably enhance model parameters. However, this burgeoning field is fragmented and lacks a common formalism, particularly concerning the ambiguous role of the reward signal -- is it a transient heuristic or a durable learning target? This paper resolves this ambiguity by introducing a unified framework that deconstructs search algorithms into three core components: the \emph{Search Mechanism}, \emph{Reward Formulation}, and \emph{Transition Function}. We establish a formal distinction between transient \textbf{Search Guidance} for TTS and durable \textbf{Parametric Reward Modeling} for Self-Improvement. Building on this formalism, we introduce a component-centric taxonomy, synthesize the state-of-the-art, and chart a research roadmap toward more systematic progress in creating autonomous, self-improving agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。