用过程奖励模型引导推理,让AI更聪明地自我修正。
PRISM: Pushing the Frontier of Deep Think via Process Reward Model-Guided Inference
- 用步骤级验证构建能量场,智能筛选和优化推理路径
- 在多个数学科学任务上达90%准确率,超越大模型基线
- 适合需要深度推理的复杂问题,尤其适合算力有限场景
深度思考方法通过生成、优化和聚合多条候选解来提升推理能力,但在推理过程中缺乏可靠的正确性信号,导致越深入思考错误越放大,正确少数解被压制,额外算力回报递减。本文提出PRISM,一种基于过程奖励模型(PRM)的推理算法,利用步骤级验证指导种群优化与解聚合。在优化阶段,将候选解视为位于PRM定义的能量场中的粒子,通过得分引导重采样和随机优化,集中概率质量于高质量推理路径,同时保持多样性。在数学与科学基准测试中,PRISM在gpt-oss-20b下分别达到AIME25 90.0%、HMMT25 75.4%、GPQA Diamond 71.4%的准确率,性能媲美甚至超过gpt-oss-120b。分析表明,PRISM在优化中持续产生方向性修正,即使初始解中正确解极少仍保持可靠,且常处于算力-精度帕累托前沿。
原文摘要 · Abstract (English)
DEEPTHINK methods improve reasoning by generating, refining, and aggregating populations of candidate solutions, which enables strong performance on complex mathematical and scientific tasks. However, existing frameworks often lack reliable correctness signals during inference, which creates a population-enhancement bottleneck where deeper deliberation amplifies errors, suppresses correct minority solutions, and yields weak returns to additional compute. In this paper, we introduce a functional decomposition of DEEPTHINK systems and propose PRISM, a Process Reward Model (PRM)-guided inference algorithm that uses step-level verification to guide both population refinement and solution aggregation. During refinement, PRISM treats candidate solutions as particles in a PRM-defined energy landscape and reshapes the population through score-guided resampling and stochastic refinement, which concentrates probability mass on higher-quality reasoning while preserving diversity. Across mathematics and science benchmarks, PRISM is competitive with or outperforms existing DEEPTHINK methods, reaching 90.0%, 75.4%, and 71.4% with gpt-oss-20b on AIME25, HMMT25, and GPQA Diamond, respectively, while matching or exceeding gpt-oss-120b. Additionally, our analysis shows that PRISM produces consistent net-directional correction during refinement, remains reliable when the initial population contains few correct candidates, and often lies on the compute-accuracy Pareto frontier.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。