通过自信度分层验证,提升大模型推理的覆盖率与准确性。
Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification

- 根据置信度高低分层处理:高置信用探索奖励防多样性坍塌,低置信由验证器筛选伪标签。
- 在6个基准上实现平均Pass@1提升9.8%,Pass@16提升18.7%。
- 适合需要强推理能力且无标注数据的场景,如复杂问题求解。
测试时强化学习已成为无需标签提升大语言模型复杂推理能力的有前景范式。尽管现有研究聚焦于Pass@1性能,优化Pass@k在无标签环境下仍被忽视却至关重要,因其衡量生成覆盖范围以支持持续探索。在无标签设置中优化Pass@k极具挑战性,因直接应用针对RLVR设计的Pass@k优势策略效果不佳。通过深入实证分析,我们发现根本原因在于:低置信样本的伪标签估计极易出错,而高置信样本的候选答案则存在严重多样性坍塌。为克服这些障碍,我们提出TTRL-CoCoV(基于自信度条件验证的测试时强化学习),一种自适应框架,可扩大Pass@k覆盖范围并提升Pass@1表现。基于关键洞察——验证能力通常领先于生成能力,TTRL-CoCoV采用置信度分层机制:对高置信样本,利用验证器进行自举并施加增强探索的奖励以防止多样性坍塌;对低置信样本,将伪标签选择交由验证器过滤错误标签;对中等置信样本,则跳过验证。大量实验表明,TTRL-CoCoV在6个广泛认可的基准上超越最佳对比方法,相比TTRL实现平均绝对提升+9.8%(Pass@1)和+18.7%(Pass@16),甚至在多个推理基准上相较完全监督的强化学习方法实现高达+5.0%的绝对提升。
原文摘要 · Abstract (English)
Test-time reinforcement learning has emerged as a promising paradigm for enhancing the complex reasoning abilities of large language models in a completely label-free manner. Despite existing studies focusing on Pass@1 performance, optimizing Pass@k remains under-explored yet critical in label-free settings, which measures generation coverage for sustained exploration. Optimizing Pass@k in label-free setting is highly non-trivial, as directly applying the Pass@k advantage designs effective for RLVR yields unsatisfactory performance. Through in-depth empirical analysis, we discover the root causes hindering performance: pseudo-label estimations for low-confidence samples have a high probability of being incorrect, while candidate answers for high-confidence samples suffer from severe diversity collapse. To overcome these hurdles, we propose TTRL-CoCoV (Test-Time Reinforcement Learning with Confidence-Conditioned Verification), a novel confidence-adaptive framework that expands Pass@k coverage and improves Pass@1 performance. Based on our key insight that verification capability generally leads generation capability, TTRL-CoCoV employs a confidence-conditioned mechanism: for high-confidence samples, it bootstraps verifier and applies an exploration-enhancing reward to prevent diversity collapse; for low-confidence samples, it delegates pseudo-label selection to the verifier to filter incorrect pseudo-labels; and for medium-confidence samples, it bypasses verification entirely. Extensive experiments demonstrate that TTRL-CoCoV outperforms the best competing methods across 6 widely-recognized benchmarks, achieves average absolute gains of +9.8% in Pass@1 and +18.7% in Pass@16 over TTRL, and even achieves absolute Pass@1 improvements of up to +5.0% across multiple reasoning benchmarks when compared against fully supervised RL methods. Our code repository: https://github.com/shanjf666/CoCoV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。