DINOv2通过自蒸馏与多裁剪增强，实现无需标签的图像预训练新突破。
Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops
- 采用自蒸馏+均值教师机制，利用多尺度图像裁剪提升特征学习。
- 在多个下游任务上超越弱监督方法，达到当前自监督学习新高度。
- 适合研究视觉表征学习、模型压缩与迁移学习的科研人员参考。
自监督学习（SSL）的最新进展使得学习通用视觉特征成为可能，这些特征既能捕捉图像的高层语义，又能保留细粒度的空间结构。其中，DINOv2 通过多裁剪视图增强和自蒸馏均值教师机制，显著超越了如 OpenCLIP 等弱监督方法（WSL），在多数基准测试中达到新纪录。本文深入分析其核心思想，追溯此前相关工作的发展脉络，并对比 DINO 与 DINOv2 在多种下游任务上的表现。研究发现，基于 Transformer 的模型在训练后展现出若干令人瞩目的涌现特性。最后，文章简要讨论了 DINOv2 的局限性、影响及未来研究方向。
原文摘要 · Abstract (English)
Recent advances in self-supervised learning (SSL) have made it possible to learn general-purpose visual features that capture both the high-level semantics and the fine-grained spatial structure of images. Most notably, the recent DINOv2 has established a new state of the art by surpassing weakly supervised methods (WSL) like OpenCLIP on most benchmarks. In this survey, we examine the core ideas behind its approach, multi-crop view augmentation and self-distillation with a mean teacher, and trace their development in previous work. We then compare the performance of DINO and DINOv2 with other SSL and WSL methods across various downstream tasks, and highlight some remarkable emergent properties of their learned features with transformer backbones. We conclude by briefly discussing DINOv2's limitations, its impact, and future research directions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。