通过反思式搜索提升推理效率,减少冗余计算。
PRISM-MCTS: Learning from Reasoning Trajectories with Metacognitive Reflection
- 引入动态共享记忆与过程奖励模型,复用过往推理经验。
- 仅需一半轨迹数即可在GPQA上超越现有方法。
- 适合资源受限场景下的高效推理任务,如低样本问答。
推理模型的兴起,如OpenAI o1,标志着认知从直觉转向深思熟虑,使扩展规律从预训练范式转向测试时计算。尽管蒙特卡洛树搜索(MCTS)在此领域展现出潜力,但现有方法通常将每次采样视为独立轨迹,缺乏信息共享,导致严重效率低下和计算冗余。为此,我们提出PRISM-MCTS,一种受人类并行思考与反思机制启发的新推理框架。该框架结合过程奖励模型(PRM)与动态共享记忆,捕捉“启发式”与“谬误”。通过强化成功策略、剪枝错误分支,实现有效优化。此外,我们设计了一种数据高效的PRM训练策略,在少样本条件下仍能实现高保真评估。在多个推理基准上的实证评估验证了其有效性:在GPQA上轨迹需求减半,性能超过MCTS-RAG与Search-o1,表明其通过有策略的推理而非盲目遍历实现推理规模扩展。
原文摘要 · Abstract (English)
PRISM-MCTS: Learning from Reasoning Trajectories with Metacognitive Reflection Siyuan Cheng, Bozhong Tian, Yanchao Hao, Zheng Wei Published: 06 Apr 2026, Last Modified: 06 Apr 2026 ACL 2026 Findings Conference, Area Chairs, Reviewers, Publication Chairs, Authors Revisions BibTeX CC BY 4.0 Keywords: Efficient/Low-Resource Methods for NLP, Generation, Question Answering Abstract: The emergence of reasoning models, exemplified by OpenAI o1, signifies a transition from intuitive to deliberative cognition, effectively reorienting the scaling laws from pre-training paradigms toward test-time computation. While Monte Carlo Tree Search (MCTS) has shown promise in this domain, existing approaches typically treat each rollout as an isolated trajectory. This lack of information sharing leads to severe inefficiency and substantial computational redundancy, as the search process fails to leverage insights from prior explorations. To address these limitations, we propose PRISM-MCTS, a novel reasoning framework that draws inspiration from human parallel thinking and reflective processes. PRISM-MCTS integrates a Process Reward Model (PRM) with a dynamic shared memory, capturing both "Heuristics" and "Fallacies". By reinforcing successful strategies and pruning error-prone branches, PRISM-MCTS effectively achieves refinement. Furthermore, we develop a data-efficient training strategy for the PRM, achieving high-fidelity evaluation under a few-shot regime. Empirical evaluations across diverse reasoning benchmarks substantiate the efficacy of PRISM-MCTS. Notably, it halves the trajectory requirements on GPQA while surpassing MCTS-RAG and Search-o1, demonstrating that it scales inference by reasoning judiciously rather than exhaustively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。