用预训练模型教学生识别失真图像，提升鲁棒性。
Distilling Vision Transformers for Distortion-Robust Representation Learning
- 教师看清晰图，学生看失真图，通过多层级知识蒸馏学习。
- 在多种失真下分类任务表现优于现有方法，仅需少量人工标注。
- 适合图像失真严重但标签稀缺的下游任务应用。
自监督学习在干净数据上已取得显著成效，但在缺乏干净观测时仍面临挑战。本文表明，可利用预训练视觉变压器模型学习对失真鲁棒的表征，并有效应用于处理失真观测的下游任务。我们提出一种非对称知识蒸馏框架：教师和学生均从同一预训练视觉变压器初始化，但接收不同图像视图——教师处理清晰图像，学生处理其失真版本。引入多层级蒸馏，对齐全局嵌入、块级特征与注意力图，证明学生虽未直接接触清晰数据，仍能近似其表征。我们在多个数据集及多种失真条件下评估该方法，在图像分类任务中始终优于同类方法，且所需人工标注量相同。
原文摘要 · Abstract (English)
Self-supervised learning has achieved remarkable success in learning visual representations from clean data, yet remains challenging when clean observations are sparse or not available at all. In this paper, we demonstrate that pretrained vision models can be leveraged to learn distortion-robust representations, which can then be effectively applied to downstream tasks operating on distorted observations. In particular, we propose an asymmetric knowledge distillation framework in which both teacher and student are initialized from the same pretrained Vision Transformer but receive different views of each image: the teacher processes clean images, while the student sees their distorted versions. We introduce multi-level distillation that aligns global embeddings, patch-level features, and attention maps and show that the student is able to approximate clean-image representations despite never directly accessing clean data. We evaluate our approach on image classification tasks across several datasets and under various distortions, consistently outperforming existing alternatives for the same amount of human supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。