将自蒸馏从图像级升级为物体级，提升视觉预训练效果
Object-level Self-Distillation for Vision Pretraining
- 以物体为单位进行自蒸馏，利用遮罩注意力聚焦目标区域
- 在ImageNet1k上达到82.6%的k-NN准确率（ViT-Large）
- 适合追求高精度视觉表征的模型开发者
当前先进的视觉预训练方法依赖于以物体为中心的数据集（如ImageNet）中的图像级自蒸馏，隐含假设每张图像仅包含单一物体。但许多ImageNet图像实际上包含多个物体，且该假设限制了向更贴近真实世界复杂性的场景级数据集的扩展。为此，本文提出物体级自蒸馏（ODIS），将自蒸馏粒度从整图改为单个物体。通过物体感知裁剪和掩码注意力机制，ODIS分离出物体特定区域，引导Transformer关注语义有意义的内容，将复杂的场景级任务分解为更简单的物体级子任务。实验表明，该方法在图像和补丁级别均提升了视觉表示能力。在推理时使用掩码，该方法在ImageNet1k上实现了82.6%的k-NN准确率（ViT-Large）。
原文摘要 · Abstract (English)
State-of-the-art vision pretraining methods rely on image-level self-distillation from object-centric datasets such as ImageNet, implicitly assuming each image contains a single object. This assumption does not always hold: many ImageNet images already contain multiple objects. Further, it limits scalability to scene-centric datasets that better mirror real-world complexity. We address these challenges by introducing Object-level Self-DIStillation (ODIS), a pretraining approach that shifts the self-distillation granularity from whole images to individual objects. Using object-aware cropping and masked attention, ODIS isolates object-specific regions, guiding the transformer toward semantically meaningful content and transforming a noisy, scene-level task into simpler object-level sub-tasks. We show that this approach improves visual representations both at the image and patch levels. Using masks at inference time, our method achieves an impressive $82.6\%$ $k$-NN accuracy on ImageNet1k with ViT-Large.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。