通过两阶段投票机制，让大模型摆脱错误主流答案的干扰。
Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism
- 先锚定主流答案，再主动生成多样辅助信号。
- 在8个基准上超越多数投票，训练更稳定。
- 无需外部监督，适合无标签推理场景。
当前无标签强化学习视觉推理（RLVR）方法如TTRL和Self-reward，在提升大语言模型复杂推理能力方面已展现出效果。然而，这些方法严重依赖伪标签估计精度，易收敛到错误但流行的答案，陷入主导模式，限制进一步提升。为此，我们提出双共识强化学习（DCRL），一种新型自监督训练方法，通过两阶段共识机制生成更可靠的训练信号。模型先作为锚点生成主流响应，再作为探索者通过临时遗忘过程产生多样化辅助信号。最终训练目标由这两组信号的调和平均决定。该过程完全不依赖外部模型或监督。在八个不同领域的基准测试中，DCRL持续优于多数投票的Pass@1指标，且训练动态更稳定。结果表明，DCRL为无标签环境下实现更强推理提供了可扩展路径。
原文摘要 · Abstract (English)
Current label-free RLVR approaches for large language models (LLMs), such as TTRL and Self-reward, have demonstrated effectiveness in improving the performance of LLMs on complex reasoning tasks. However, these methods rely heavily on accurate pseudo-label estimation and converge on spurious yet popular answers, thereby trapping in a dominant mode and limiting further improvements. Building on this, we propose Dual Consensus Reinforcement Learning (DCRL), a novel self-supervised training method which is capable of generating more reliable learning signals through a two-stage consensus mechanism. The model initially acts as an anchor, producing dominant responses; then it serves as an explorer, generating diverse auxiliary signals via a temporary unlearning process. The final training target is derived from the harmonic mean of these two signal sets. Notably, the process operates entirely without external models or supervision. Across eight benchmarks and diverse domains, DCRL consistently improves Pass@1 over majority vote while yielding more stable training dynamics. These results demonstrate that DCRL establishes a scalable path toward stronger reasoning without labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。