arXiv:2607.13643cs.LGcs.AI2026-07被引 1

利用共识信息提升无标签大模型推理能力,效果接近有标签训练。

Consensus as Privileged Context for Label-Free Self-Distillation

  • 通过多数答案锚定教师模型,实现逐标记的密集监督。
  • 数学与科学推理任务上准确率提升12点,优于强化学习方法6点。
  • 适合无标注数据训练,且能跨任务迁移,逼近有标签效果。

采样多个解并返回多数答案是无需标签即可提升大语言模型推理准确性的可靠方法。现有方法仅将共识用于有限形式:作为筛选微调样本的过滤器、答案间的偏好判断或强化学习的标量奖励,忽略了多数解所包含的丰富信息。本文提出CANON(Consensus-ANchored self-distillatiON),一种无标签训练方法,将共识转化为密集的、逐标记级别的监督信号。对每个未标注提示,CANON采样多个解,提取多数答案,并将冻结的模型快照条件于达到该答案的解;此共识锚定的教师模型在每一步标记上监督自身生成过程。在数学与科学推理基准测试中,CANON使pass@1最高提升12点,相比无标签强化学习提升6点,且仅需其七分之一计算量,并接近使用真实答案训练的教师模型表现。在合并的无标签数据上训练后,模型可迁移至未见基准,性能匹配使用真实标签的训练方法。分析表明,提升并非单纯分布锐化:训练后,模型在32次尝试中首次解决此前从未解决的问题,且其多数投票本身也更准确。

原文摘要 · Abstract (English)

Sampling multiple solutions and returning the majority answer is among the most reliable ways to improve the reasoning accuracy of large language models without labels, and a growing family of methods converts this consensus signal into training supervision. However, existing approaches use consensus only in restricted forms: as a filter that selects solutions for fine-tuning, as a preference between answers, or as a scalar reward for reinforcement learning, discarding most of the information that the agreeing solutions contain. We present CANON (Consensus-ANchored self-distillatiON), a label-free training method that turns consensus into dense, token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, extracts the majority answer, and conditions a frozen snapshot of the model on a solution that reaches it; this consensus-anchored teacher then supervises the model on its own rollouts at every token. Experiments on mathematical and scientific reasoning benchmarks show that CANON improves pass@1 by up to 12 points, outperforming label-free reinforcement learning by 6 points at a seventh of its compute and approaching a teacher conditioned on gold solutions; trained on pooled unlabeled data, it transfers to held-out benchmarks, matching training methods that use gold labels. Analysis suggests that the improvements are not pure distribution sharpening: after training, the model solves problems it previously never solved in 32 attempts, and its majority vote itself becomes more accurate.

自蒸馏无标签训练推理增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。