通过共识门控实现无标签场景下的知识蒸馏，提升模型泛化能力。
GATES: Self-Distillation under Privileged Context with Consensus Gating
- 基于多个推理路径的共识生成可靠性信号，动态控制学习
- 在无文档环境下准确率从46.0%提升至62.0%，数学基准平均提升至35.4%
- 适用于缺乏标注数据的复杂问答任务，尤其适合自蒸馏场景
我们研究在监督不可靠场景下的自蒸馏方法：无真实标签、可验证奖励或外部评分器来评估答案。聚焦于上下文不对称的文档增强型问答任务，单一模型在训练时作为有文档访问权限的教师，在测试时作为仅凭问题作答的学生。不假设教师正确性，而是通过采样多个基于文档的推理路径，利用一致性生成在线监督信号，并以该可靠性信号为门控机制进行知识蒸馏。基于此，我们蒸馏完整的教师推理轨迹（而非仅最终答案），提供密集且稳定的训练信号。实验表明，这种共识门控的轨迹蒸馏显著提升了学生模型在无文档场景下的迁移性能。在保留的域内评估中，准确率从46.0%提升至62.0%；在公开的无文档数学基准上，平均（maj@8）准确率从20.2%提升至35.4%。
原文摘要 · Abstract (English)
We study self-distillation in settings where supervision is unreliable: there are no ground truth labels, verifiable rewards, or external graders to evaluate answers. We focus on document-grounded question answering with asymmetric context, where a single model serves as both tutor (with access to a relevant source document during training) and student (answering from the question alone at test time). Rather than assuming tutor correctness, we derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces and using agreement to gate learning. Conditioned on this reliability signal, we distill knowledge through full tutor reasoning trajectories (not just final answers), providing a dense and stable learning signal. Empirically, this consensus-gated trajectory distillation substantially improves transfer to the document-free student. Held-out in-domain accuracy under asymmetric evaluation improves from 46.0\% to 62.0\%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2\% to 35.4\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。