OPV通过验证长思维链的总结结果,高效准确地检测推理错误。
OPV: Outcome-based Process Verifier for Efficient Long Chain-of-Thought Verification
- 基于结果反推过程,用迭代主动学习提升验证能力
- 在新基准上达83.1的F1分数,优于大模型如Qwen3-Max-Preview
- 适合需要低成本大规模标注的复杂推理验证场景
大型语言模型在复杂推理任务中取得进展,得益于可验证奖励的强化学习(RLVR),这离不开可靠的验证机制。然而,现有基于结果的验证器(OVs)无法检查长思维链中的不可靠中间步骤;而基于过程的验证器(PVs)因高质量标注稀缺,难以可靠检测复杂长思维链中的错误。为此,我们提出基于结果的过程验证器(OPV),通过验证长思维链的总结结果来实现高精度、高效率的验证,并支持大规模标注。为增强验证能力,采用迭代主动学习框架,结合专家标注,逐步提升OPV性能,每轮仅标注最不确定样本,通过拒绝微调(RFT)和RLVR训练新版本。大量实验表明,OPV在自建基准OPV-Bench上达到新最佳表现,F1达83.1,优于更大模型Qwen3-Max-Preview的76.3。同时有效识别合成数据中的假阳性,与专家评估高度一致。与策略模型协作时持续提升性能,例如在AIME2025上将DeepSeek-R1-Distill-Qwen-32B准确率从55.2%提升至73.3%。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved significant progress in solving complex reasoning tasks by Reinforcement Learning with Verifiable Rewards (RLVR). This advancement is also inseparable from the oversight automated by reliable verifiers. However, current outcome-based verifiers (OVs) are unable to inspect the unreliable intermediate steps in the long reasoning chains of thought (CoTs). Meanwhile, current process-based verifiers (PVs) have difficulties in reliably detecting errors in the complex long CoTs, limited by the scarcity of high-quality annotations due to the prohibitive costs of human annotations. Therefore, we propose the Outcome-based Process Verifier (OPV), which verifies the rationale process of summarized outcomes from long CoTs to achieve both accurate and efficient verification and enable large-scale annotation. To empower the proposed verifier, we adopt an iterative active learning framework with expert annotations to progressively improve the verification capability of OPV with fewer annotation costs. Specifically, in each iteration, the most uncertain cases of the current best OPV are annotated and then subsequently used to train a new OPV through Rejection Fine-Tuning (RFT) and RLVR for the next round. Extensive experiments demonstrate OPV's superior performance and broad applicability. It achieves new state-of-the-art results on our held-out OPV-Bench, outperforming much larger open-source models such as Qwen3-Max-Preview with an F1 score of 83.1 compared to 76.3. Furthermore, OPV effectively detects false positives within synthetic dataset, closely align with expert assessment. When collaborating with policy models, OPV consistently yields performance gains, e.g., raising the accuracy of DeepSeek-R1-Distill-Qwen-32B from 55.2% to 73.3% on AIME2025 as the compute budget scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。