无需外部监督，通过模型自身推理分歧实现更稳定的语言模型推理。
Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrast
- 从学生模型的未标注推理路径中挖掘共识与分歧信号，构建无监督教师。
- 共识引导知识迁移，少数异常路径通过对比惩罚避免错误固化。
- 适用于数学推理等高阶任务，尤其适合缺乏标注数据的场景。
基于策略的自蒸馏通过让学生在实际访问的状态上查询教师来提升语言模型的推理能力。现有方法通过向教师提供特权上下文建立信息不对称，但依赖外部监督（如正确答案或验证器）来构造这种优势。本文提出完全无监督的CoDA（共识与分歧对齐）框架，从模型自身未标注轨迹的隐含不确定性结构中生成可靠特权信息。CoDA提取两个互补信号：正向分支中，答案级共识识别出稳定的推理模式，用于冻结自教师对新学生轨迹提供密集分布指导；但一致性不等于正确性，仅靠正向蒸馏可能放大相关错误形成虚假共识。为此，CoDA引入负向分支利用分歧——少数轨迹被视为不稳定备选方案，通过参考锚定的KTO风格校准目标进行温和惩罚。该无配对二元反馈提供鲁棒正则化，无需假设共识即绝对真值。在竞赛级数学基准上的实证评估表明，CoDA显著提升推理性能，优于自生成基线，并有效防止错误共识导致的训练失稳。
原文摘要 · Abstract (English)
On-policy self-distillation improves language-model reasoning by querying a teacher on states actually visited by the student. Recent methods create a powerful information asymmetry by exposing the teacher to privileged context, yet they fundamentally rely on external supervision---such as gold solutions or verifiers---to construct this advantage. We introduce CoDA (Consensus and Disagreement Alignment), a fully unsupervised framework that creates reliable privileged information entirely from the latent uncertainty structure of a model's own unlabeled rollouts. CoDA extracts two complementary signals. In the positive branch, answer-level consensus identifies a stable reasoning mode, which conditions a frozen self-teacher to provide dense distributional guidance on fresh student trajectories. However, because agreement does not guarantee correctness, positive-only distillation risks amplifying correlated errors into a false consensus. To break this harmful feedback loop, CoDA incorporates a negative branch that exploits disagreement: minority trajectories are treated as unstable alternatives and gently penalized via a reference-anchored, KTO-style calibration objective. This unpaired binary feedback provides robust regularization without requiring the strong assumption that the consensus is the absolute ground truth. Empirical evaluations on competition-level mathematical benchmarks demonstrate that CoDA significantly improves reasoning, outperforming self-generated baselines and effectively stabilizing training against erroneous consensus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。