arXiv:2603.04304cs.CL2026-03被引 8

通过成对自验证提升推理模型生成质量,效率更高且准确率显著提升。

$V_1$: Unifying Generation and Self-Verification for Parallel Reasoners

  • 用成对排序统一生成与验证,动态分配计算资源到最不确定的候选对。
  • 在代码和数学推理任务中,相比点对点验证提升10%准确率,效率更高。
  • 适合需要高精度推理的复杂任务,如编程、数学证明等场景。

复杂推理任务的测试时扩展表明,通过独立采样并聚合多个解法来利用推理时计算资源,能显著提升任务表现。然而,验证是关键瓶颈:采样有效前提是能可靠识别正确解。现有方法通常使用标量评分独立评估候选解,我们发现模型在成对自验证方面强得多。为此提出 $V_1$ 框架,通过高效的成对排名统一生成与验证。$V_1$ 包含两个组件:$V_1$-Infer,一种基于不确定性的锦标赛式排序算法,动态将自验证计算分配给相对正确性最不确定的候选对;$V_1$-PairRL,一种联合训练生成器与成对自验证器的强化学习框架,确保验证器适应生成器分布变化。在 LiveCodeBench、CodeContests、SWE-Bench(代码生成)及 AIME、HMMT(数学推理)基准上,$V_1$-Infer 相比点对点验证提升 Pass@1 最多 10%,优于近期测试时扩展方法且更高效。$V_1$-PairRL 在测试时扩展上相较标准 RL 和点对点联合训练提升 7–9%,并在代码生成任务中使基础 Pass@1 提升最高达 8.7%。

原文摘要 · Abstract (English)

Test-time scaling for complex reasoning tasks shows that leveraging inference-time compute, by methods such as independently sampling and aggregating multiple solutions, results in significantly better task outcomes. However, a critical bottleneck is verification: sampling is only effective if correct solutions can be reliably identified among candidates. While existing approaches typically evaluate candidates independently via scalar scoring, we demonstrate that models are substantially stronger at pairwise self-verification. Leveraging this insight, we introduce $V_1$, a framework that unifies generation and verification through efficient pairwise ranking. $V_1$ comprises two components: $V_1$-Infer, an uncertainty-guided algorithm using a tournament-based ranking that dynamically allocates self-verification compute to candidate pairs whose relative correctness is most uncertain; and $V_1$-PairRL, an RL framework that jointly trains a single model as both generator and pairwise self-verifier, ensuring the verifier adapts to the generator's evolving distribution. On code generation (LiveCodeBench, CodeContests, SWE-Bench) and math reasoning (AIME, HMMT) benchmarks, $V_1$-Infer improves Pass@1 by up to $10%$ over pointwise verification and outperforms recent test-time scaling methods while being significantly more efficient. Furthermore, $V_1$-PairRL achieves $7$--$9%$ test-time scaling gains over standard RL and pointwise joint training, and improves base Pass@1 by up to 8.7% over standard RL in a code-generation setting.

推理增强自验证代码生成强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。