用视觉对比提升自蒸馏效果,无需外部教师或额外信息。
Visual Contrastive Self-Distillation

- 通过图像内容消融与原图对比,生成自蒸馏信号。
- 在2B至8B模型上,多任务平均准确率提升4.77%~3.75%。
- 无需外部教师、特权答案或推理轨迹,部署更简单。
在策略自蒸馏(OPSD)中,传统方法依赖外部教师或不对称信息来确保教师提供更强学习信号。本文提出视觉对比自蒸馏(VCSD),仅通过输入条件驱动,将图像内容移除转化为自蒸馏信号。在每个学生生成的响应前缀处,基于动量平均(EMA)的教师在相同提示和前缀下生成两个下一词分布:一个基于原始图像,另一个基于内容擦除的控制图像。两者的词级对数概率差,突出那些因实例级视觉内容而概率显著提升的候选词。利用该对比,锐化教师在合理支持范围内的原图分布,并将完整分布目标蒸馏至学生。在ViRL39K数据集上,VCSD在Qwen3-VL和Qwen3.5模型上均优于对照组。例如,在Qwen3-VL模型中,2B、4B、8B版本的七项基准平均准确率分别从62.27%→67.04%、71.30%→73.16%、72.51%→76.26%。此外,VCSD无需外部教师、特权答案、视觉证据信号、推理轨迹或额外推理成本。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yielding a simpler form of OPSD driven purely by input conditioning. For this purpose, we propose Visual Contrastive Self-Distillation, namely VCSD, which converts image-content removal into an on-policy self-distillation signal. At each student-generated response prefix, the EMA teacher produces two next-token distributions under the same prompt and prefix -- one conditioned on the original image and the other on a content-erased control. Their token-wise log-probability difference highlights candidates whose likelihood is specifically increased by the instance-level visual content. We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student. Using ViRL39K dataset, VCSD consistently outperforms matched OPSD across Qwen3-VL and Qwen3.5 models. For example, on Qwen3-VL, it improves the seven-benchmark aggregate from $62.27\% \rightarrow 67.04\%$ at 2B, $71.30\% \rightarrow 73.16\%$ at 4B, and $72.51\% \rightarrow 76.26\%$ at 8B. Furthermore, VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。