arXiv:2607.28457cs.AIcs.CL2026-07

让大模型自己判断答案对错,动态决定是否继续思考。

SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute

论文配图:SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute
图 1 · 摘自论文原文
  • 用自验证机制控制思考次数,避免无意义计算。
  • 平均仅需2.99轮推理,数学题准确率达56.3%。
  • 无需外部评分,适合资源受限的推理场景。

提升语言模型推理能力可通过增加测试时计算量实现,但固定预算会浪费算力于简单问题,而依赖外部反馈的验证方法存在局限。本文提出自验证精炼(SVR),一种无需外部监督的多轮强化学习框架,学习以自验证作为计算控制策略。每轮中,模型生成答案、正确性判断和置信度;仅当判断为“正确”且置信度超过阈值时保留答案,否则继续自我修正。真实正确性仅用于训练奖励构建,不参与推理过程。在七个数学推理基准上,使用Qwen3.5-2B模型,SVR平均仅需2.99次推理,宏平均准确率达0.563。相比标准GRPO、强基线及固定预算的有监督参考模型,其性能更优且推理轮次显著减少。结果表明,可学习的自验证能有效作为内部信号,指导答案保留与动态计算分配。

原文摘要 · Abstract (English)

Scaling test-time computation can improve language-model reasoning, but uniform budgets waste computation on easy inputs, while verifier-guided refinement relies on external feedback. We introduce Self-Verifying Refinement (SVR), an oracle-free multi-turn reinforcement learning framework that learns to use self-verification as a compute-control policy. At each turn, the model produces a solution together with a discrete correctness verdict and a confidence score; it retains the current answer only when the verdict is Correct and confidence exceeds a threshold, and otherwise continues refinement using its own self-verification. Ground-truth correctness is used only to construct training rewards and is never exposed to the policy through refinement prompts or required at inference. SVR is trained with GRPO on fixed-horizon trajectories using rewards that promote solution correctness, calibration-aware self-verification, and stop-ready correct states; adaptive stopping is activated only at inference. On seven mathematical reasoning benchmarks with Qwen3.5-2B, SVR achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average. In the evaluated complete-system comparison, it exceeds standard GRPO, strong multi-turn baselines, and a fixed-budget oracle-guided score-feedback reference while requiring substantially fewer turns than fixed ten-turn inference. These results demonstrate that learned self-verification can serve as an effective internal control signal for answer retention and adaptive test-time compute allocation.

推理优化强化学习自验证动态计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。