用树搜索整合多个模型推理步骤，提升复杂问题求解能力。
Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning
- 将多模型推理过程建模为马尔可夫决策过程，动态选择最优推理路径。
- 在MATH和MQA数据集上分别提升3.6%和4.3%准确率。
- 适合需要高精度逻辑推理的数学与认知任务场景。
尽管大语言模型取得进展，开源模型在复杂推理任务中仍表现不稳定。现有集成方法在词元或输出层面均难以解决此问题。为此，我们提出语言模型蒙特卡洛树搜索集成框架（LE-MCTS），实现语言模型在推理过程层面的集成。该框架将多模型逐步推理建模为马尔可夫决策过程：状态代表中间推理路径，动作是从预定义池中选择一个语言模型生成下一步推理。基于过程奖励模型，LE-MCTS在不同模型生成的推理路径上进行树搜索，识别最准确的推理链。在五个数学推理基准上的实验表明，该方法优于单模型解码与传统集成方法。特别地，在MATH和MQA数据集上分别提升3.6%和4.3%，证明其在解决复杂推理问题中的有效性。
原文摘要 · Abstract (English)
Despite recent advances in large language models, open-source models often struggle to consistently perform well on complex reasoning tasks. Existing ensemble methods, whether applied at the token or output levels, fail to address these challenges. In response, we present Language model Ensemble with Monte Carlo Tree Search (LE-MCTS), a novel framework for process-level ensembling of language models. LE-MCTS formulates step-by-step reasoning with an ensemble of language models as a Markov decision process. In this framework, states represent intermediate reasoning paths, while actions consist of generating the next reasoning step using one of the language models selected from a predefined pool. Guided by a process-based reward model, LE-MCTS performs a tree search over the reasoning steps generated by different language models, identifying the most accurate reasoning chain. Experimental results on five mathematical reasoning benchmarks demonstrate that our approach outperforms both single language model decoding algorithms and language model ensemble methods. Notably, LE-MCTS improves performance by 3.6% and 4.3% on the MATH and MQA datasets, respectively, highlighting its effectiveness in solving complex reasoning problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。