用博弈式搜索提升大模型解奥数题能力，比传统方法更快更准。
LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning
- 结合蒙特卡洛树搜索与自我修正，动态优化解题路径。
- 在奥数题集上达到87.3%正确率，优于ToT和rStar方法。
- 适合需要高阶逻辑推理的研究者或竞赛级数学任务。
本文提出一种先进的数学问题求解框架LLaMA-Berry，用于提升大语言模型的数学推理能力。该框架将蒙特卡洛树搜索（MCTS）与迭代式自我修正（Self-Refine）结合，优化推理路径，并引入成对奖励模型（PPRM）从全局评估不同路径。通过利用大模型的自评与重写能力，基于自我修正的蒙特卡洛树搜索（SR-MCTS）克服了传统逐步贪心搜索效率低、探索不足的问题。成对偏好奖励模型（PPRM）借鉴人类反馈强化学习（RLHF），采用增强型博达计数（EBC）方法整合成对偏好，生成全局排序得分以发现更优解，有效缓解数学推理任务中评分波动与非独立分布问题。该框架在通用及高级基准上测试，相比ToT和rStar等方法，在搜索效率与解题能力上表现更优，尤其在复杂奥数基准如GPQA、AIME24和AMC23上展现出显著优势。
原文摘要 · Abstract (English)
This paper presents an advanced mathematical problem-solving framework, LLaMA-Berry, for enhancing the mathematical reasoning ability of Large Language Models (LLMs). The framework combines Monte Carlo Tree Search (MCTS) with iterative Self-Refine to optimize the reasoning path and utilizes a pairwise reward model to evaluate different paths globally. By leveraging the self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS (SR-MCTS) overcomes the inefficiencies and limitations of conventional step-wise and greedy search algorithms by fostering a more efficient exploration of solution spaces. Pairwise Preference Reward Model~(PPRM), inspired by Reinforcement Learning from Human Feedback (RLHF), is then used to model pairwise preferences between solutions, utilizing an Enhanced Borda Count (EBC) method to synthesize these preferences into a global ranking score to find better answers. This approach addresses the challenges of scoring variability and non-independent distributions in mathematical reasoning tasks. The framework has been tested on general and advanced benchmarks, showing superior performance in terms of search efficiency and problem-solving capability compared to existing methods like ToT and rStar, particularly in complex Olympiad-level benchmarks, including GPQA, AIME24 and AMC23.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。