通过生成器与验证器协同进化，解决无标签推理中的共识陷阱问题。
CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution
- 模型在生成和验证角色间交替，互相提升能力
- 数学推理准确率提升4.7%-5.9%，自验证准确率达85%以上
- 适合追求高可靠性推理的AI系统开发者
无标签强化学习使大语言模型在无真实答案监督下提升推理能力，通常通过多数投票结果作为伪标签。然而我们发现一个关键缺陷：训练过程强化自我一致性，导致输出多样性下降，模型会自信地重复系统性错误。我们称此为共识陷阱。为此提出CoVerRL框架，让单一模型在生成与验证角色间交替，彼此驱动。多数投票提供有噪声但具信息量的监督用于训练验证器，而不断改进的验证器逐步过滤伪标签中的自一致错误。这种协同进化形成良性循环，全程保持高奖励准确性。在Qwen与Llama模型族上的实验表明，CoVerRL在数学推理基准上优于无标签基线4.7%-5.9%。自验证准确率从约55%提升至85%以上，证实了双能力的真实共进化。
原文摘要 · Abstract (English)
Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。