解决数学推理中过程奖励模型过优化问题，提升搜索稳定性。
Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide
- 提出最大最小优化框架，增强对奖励噪声的鲁棒性。
- 无需微调，在16组测试中平均提升17-35%。
- 适合需要稳定推理路径的高可靠性场景。
过程奖励模型（PRM）为基于搜索的推理提供细粒度的步骤级指导，使推理时计算资源可集中于有前景的部分解。然而，近期研究表明，PRM引导的搜索可能过度优化不完善的奖励信号，剪枝有效路径并扩展虚假路径。本文从理论上揭示：直接使用PRM评分易受验证器噪声影响，产生极端值效应——随着推理深度增加，无效前缀更可能获得虚高的评分。为此，我们将PRM引导搜索建模为对合理奖励扰动的鲁棒优化问题，提出最大最小PRM引导搜索方法，实现无需训练的鲁棒过程监督。该方法在步级评分存在噪声时仍能保留有希望的备选路径，显著降低对过优化异常值的敏感性。无需微调或在线调整，最大最小搜索在16个设置中的14个上优于结果级和步级基线，平均性能提升17-35%。
原文摘要 · Abstract (English)
Process reward models (PRMs) provide dense step-level guidance for search-based reasoning, enabling inference-time compute to be allocated toward promising partial solutions. However, recent evidence suggests that PRM-guided search can over-optimize imperfect process rewards, pruning viable trajectories while expanding spurious ones. In this work, we theoretically show that directly leveraging PRM score is vulnerable to verifier noise through an extreme-value effect: non-viable prefixes become more likely to receive spuriously high scores as reasoning depth increase. Therefore, we formulate the PRM-guided search as a robust optimization problem over plausible reward perturbations, termed maximin PRM-guided search, leading to a training-free robust process supervision method that preserves promising alternatives when step-level scores are noisy. Maximin PRM-guided search mitigates this failure mode by reducing sensitivity to over-optimized PRM outliers. Without fine-tuning or online adaptation, maximin search consistently improves the PRM-guided search by 17-35\% on average, outperforming outcome- and step-level baselines in 14 out of 16 settings. Our source code is available at https://github.com/tjoo512/maximin-search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。