不用标注或强教师模型，靠图像增强差异实现高效自监督学习
Self-Supervised Visual On-Policy Distillation
- 用强增强视图与原图的不对称性构造学习信号
- 在6个细粒度感知任务上将模型准确率提升至77.4%
- 无需额外数据或标注，效果接近使用特权信息的方法
视觉在线策略蒸馏依赖于教师-学生间的有效不对称性，通常通过更强的教师模型或特权监督实现。本文提出反向思路：不给教师加信息，而是从学生中减信息。这种不对称性在无真实标注、奖励或强教师的情况下，自然生成有效的学习信号。基于此，我们提出自监督视觉在线策略蒸馏（S²VOPD），利用不对称增强视图构建在线学习信号。该方法将教师对原始图像的分布，蒸馏到学生对强增强图像的分布中。系统探索多种视觉增强设计，发现：(1) 不对称性至关重要，四类增强均提升性能，对称自蒸馏反而下降；(2) 增强强度需适中，过强或过弱皆不利；(3) 增强不能完全抹除关键信息，否则产生误导性差异。在六个细粒度感知基准上，S²VOPD将Qwen3.5-4B准确率从70.7%提升至77.4%，超越所有开源模型，接近2350亿参数的Qwen3-VL，并优于GPT-5.4。相同训练数据下，恢复了96%由特权信息带来的性能提升。
原文摘要 · Abstract (English)
Visual on-policy distillation relies heavily on an informative teacher-student asymmetry, through either a larger, stronger teacher or privileged supervision, such as reference answers or ground-truth regions of interest. This raises a fundamental question: where can informative asymmetry come from when nothing privileged is available? We answer this by inverting where the asymmetry comes from. Rather than adding privileged information to the teacher, we subtract information from the student. This asymmetry creates the same effective learning signal for free as a teacher with access to information unavailable to the student, without ground-truth annotations, rewards, or a separate stronger teacher model. Building on this principle, we introduce Self-Supervised Visual On-Policy Distillation (S$^2$VOPD), a simple yet effective method that constructs on-policy learning signals from asymmetric augmented views. S$^2$VOPD distills the teacher's distribution conditioned on the original image on-policy into the student distribution conditioned on a strongly augmented view of the same image. We systematically explore a broad design space of visual augmentations and uncover that (1) asymmetry matters: all four augmentation families improve performance, while symmetric self-distillation degrades it; (2) strength matters: performance peaks at a moderate strength; and (3) the gap must remain task-consistent: augmentations that completely remove the question-relevant evidence can induce large but uninformative discrepancies. Across six fine-grained perception benchmarks, S$^2$VOPD improves Qwen3.5-4B from 70.7% to 77.4%, above all open-source models compared, up to Qwen3-VL at 235B, and surpasses GPT-5.4. While holding training data the same, it recovers 96% of the improvement achieved by methods with privileged information. Website is at https://williamium3000.github.io/s2vopd
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。