主动获取少量真实标签,让强化学习更稳定高效。
When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards

- 选关键样本主动获取真实标签,结合伪标签训练。
- 在有限标注预算下,性能优于纯伪标签方法。
- 适合标注成本高、需稳定训练的RL场景。
大语言模型在基于可验证奖励的强化学习(RLVR)驱动下,推理能力取得显著进展。然而,RLVR依赖真实标签计算奖励,而真实标签获取成本高昂。无监督的RLVR虽尝试用伪标签替代,但易导致训练崩溃,且不同样本的标注价值差异大。本文提出主动可验证奖励强化学习(RLAVR),主动获取少数精选样本的真实标签,并与伪标签融合,从而稳定训练并提升性能。为识别有价值样本,提出校正优势差距(CAG)指标,分析样本级监督价值。进一步设计校正感知可靠性估计(CARE),将理想化的CAG准则转化为实用的查询前策略,显著提升训练稳定性。跨领域、多模型家族和多种规模的实验验证了方法的有效性与通用性。代码已开源:https://github.com/Lumina04/CARE。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable advancements in reasoning capabilities empowered by Reinforcement Learning with Verifiable Rewards (RLVR). Nonetheless, RLVR intrinsically relies on ground-truth labels for reward computation, the acquisition of which is often prohibitively expensive in real-world scenarios. While unsupervised RLVR paradigms attempt to circumvent this by training on pseudo-labels, they are notoriously susceptible to training collapse. Moreover, different samples often exhibit varying annotation values. In this paper, we propose Reinforcement Learning with Active Verifiable Rewards (RLAVR), which actively acquires ground-truth labels for a small set of selected samples and integrates them with pseudo-labels, thereby stabilizing training dynamics and improving performance under limited annotation budgets. To identify valuable samples, we propose the Corrective Advantage Gap (CAG) metric and analyze the sample-level supervision value. Building on this, we introduce Correction-Aware Reliability Estimation for RLAVR (CARE), which translates the oracle CAG criterion into a practical pre-query acquisition policy to substantially improve training stability. Extensive experiments across diverse domains, model families, and model scales demonstrate the effectiveness and generality of our approach. Our code is available at https://github.com/Lumina04/CARE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。