arXiv:2604.02621cs.CLcs.LG2026-04被引 3

用大模型当裁判,让小模型通过强化学习从无标签数据中提升推理能力。

Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge

  • 用单令牌对齐的连续思维链奖励评估学生模型推理过程。
  • 在无标签数据上训练,数学推理准确率提升5-10%。
  • 结合真实与自动生成奖励,效果更稳定且泛化更强。

强化学习显著提升语言模型的推理能力,但现有方法多依赖可验证的真值奖励和标注数据。为此,我们提出一种基于强化学习的推理知识蒸馏框架,利用连续的、由大模型生成的奖励信号。该方法通过单令牌日志概率直接计算连续思维链奖励(CCR),评估学生模型的推理轨迹,提供高效可扩展的在线训练信号,适用于大规模无标签数据。实验表明,配合强裁判模型时,仅使用CCR即可达到甚至超过真值或伪标签奖励的性能,且随着无标签数据增多优势更明显。进一步发现,将二者结合在半监督设置下具有高度协同效应:可验证奖励稳定了CCR,而CCR提升了可验证奖励在相关任务上的泛化能力。我们在多种模型架构与数据规模下进行了全面对比,结果表明该方法持续提升数学推理能力,在多个任务上实现5-10%的绝对性能提升。

原文摘要 · Abstract (English)

Reinforcement Learning (RL) substantially improves the reasoning capabilities of language models, but most existing RL fine-tuning approaches rely entirely on ground-truth verifiable rewards and thus labeled datasets with verifiable answers. To overcome this, we propose a RL framework for reasoning distillation that leverages continuous, LLM-based rewards. Our method employs an efficient mechanism that computes a continuous CoT reward (CCR) directly from a single-token logit of a judge LLM, evaluating the student model's reasoning trajectory. This formulation provides an effective and scalable online training signal that can be applied to large volumes of unlabeled data. We demonstrate that, when paired with a strong judge, simply using CCR achieves performance comparable to that of ground-truth or pseudo-label verifiable rewards, and even surpasses them as the amount of unlabeled data increases. Furthermore, we find that combining them in a semi-supervised setup is highly synergistic: verifiable rewards help stabilize the CCR, while CCR improves the generalizability of verifiable rewards to related tasks. We also provide a comprehensive empirical comparison of various reward sources across multiple model architectures and dataset sizes. Our results show that this semi-supervised approach consistently enhances mathematical reasoning, yielding an absolute improvement of 5-10% on multiple reasoning tasks.

强化学习知识蒸馏大模型半监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。