arXiv:2508.00410cs.LG2025-08中稿 · ICLR被引 12

用多视角自监督提升大模型推理能力,避免奖励欺骗。

Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models

  • 引入互补视角的自监督信号,防止单一反馈导致训练崩溃。
  • 在数学推理任务上平均提升3.31%,部分模型超人工标注强化学习效果。
  • 适合追求稳定推理训练的大模型研究者与应用开发者。

尽管基于可验证奖励的强化学习(RLVR)能有效提升大语言模型(LLM)的推理能力,但其依赖人工标注标签,面临扩展难题,尤其在复杂任务中。近期自奖励方法尝试无标签替代方案以激发模型推理能力,却常遭遇训练崩溃问题——单视角监督易形成自洽幻觉,导致奖励欺骗。受自监督学习成功启发,我们提出新框架Co-rewarding,通过引入另一视角的互补监督信号增强训练稳定性。具体实现为两种方式:(1) Co-rewarding-I 在数据侧通过语义相似问题间的对比一致性生成奖励信号;(2) Co-rewarding-II 在模型侧使用缓慢更新的伪标签教师模型进行自蒸馏。二者分别引入不同层级差异,提高训练对平凡解的抵抗能力。还探索了两者的正交组合以进一步提升性能。实验表明,Co-rewarding 在多种设置下均表现稳定,平均优于其他自奖励基线3.31%;尤其在Llama-3.2-3B-Instruct上提升7.49%。值得注意的是,其在部分任务中达到甚至超越采用真实标签(GT)的RLVR表现,例如在GSM8K上使用Qwen3-8B-Base模型获得Pass@1 94.01%,显著高于GT结果。代码已开源。

原文摘要 · Abstract (English)

While reinforcement learning with verifiable rewards (RLVR) is effective to improve the reasoning ability of large language models (LLMs), its reliance on human-annotated labels leads to the scaling up dilemma, especially for complex tasks. Recent self-rewarding methods investigate a label-free alternative to unlock the reasoning capabilities of LLMs, yet they frequently encounter the non-negligible training collapse issue, as the single-view supervision signal easily forms the self-consistent illusion, yielding the reward hacking. Inspired by the success of self-supervised learning, we propose \textit{Co-rewarding}, a novel self-supervised RL framework that improves training stability by seeking complementary supervision from another views. Specifically, we instantiate Co-rewarding in two ways: (1) \textit{Co-rewarding-I} is a data-side instantiation that derives reward signals from contrastive agreement across semantically analogous questions; and (2) \textit{Co-rewarding-II} is a model-side instantiation that maintains a slowly-updated reference teacher with pseudo labels to realize self-distillation. Intuitively, such instantiations introduce different levels of discrepancy to increase the difficulty of training collapse on trivial reasoning solutions. We also explore their orthogonally combined version to further boost the performance. Empirically, Co-rewarding exhibits stable training across various setups, and outperforms other self-rewarding baselines by $+3.31\%$ improvements on average on multiple mathematical reasoning benchmarks, especially by $+7.49\%$ on Llama-3.2-3B-Instruct. Notably, Co-rewarding reaches or even surpasses RLVR with ground-truth (GT) label in several cases, such as a Pass@$1$ of $94.01\%$ on GSM8K with Qwen3-8B-Base remarkably higher than GT. Our code is released at https://github.com/tmlr-group/Co-rewarding.

强化学习自监督推理增强大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。