arXiv:2510.20994cs.CVcs.AI2025-10NeurIPS

用无标注视频让视觉模型自适应新场景,避免知识遗忘。

VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models

  • 基于多视角物体视频,通过自蒸馏实现无监督微调。
  • 在2个数据集上显著提升分类性能,优于基线和已有方法。
  • 适合标签稀缺、分布漂移的视觉应用,如工业检测、医疗影像。

基础模型通过大规模预训练和监督微调,在多种任务中表现优异。但在分布偏移严重且标签稀少的领域,监督微调难以实施。尽管生成式语言模型常采用持续自监督学习进行适应,但该策略对视觉编码器模型效果不佳。为此,本文提出 VESSA:一种基于视频的物体中心自监督适应方法,无需标注即可在新域中微调视觉基础模型。该方法利用短时多视角物体中心视频,通过自蒸馏机制,在不遗忘预训练知识的前提下,有效学习对不同拍摄条件的鲁棒性。实验表明,3种视觉基础模型在2个数据集上均取得一致改进,显著优于基线和现有方法。代码已公开于 https://github.com/jesimonbarreto/VESSA。

原文摘要 · Abstract (English)

Foundation models have advanced computer vision by enabling strong performance across diverse tasks through large-scale pretraining and supervised fine-tuning. However, they may underperform in domains with distribution shifts and scarce labels, where supervised fine-tuning may be infeasible. While continued self-supervised learning for model adaptation is common for generative language models, this strategy has not proven effective for vision-centric encoder models. To address this challenge, we introduce a novel formulation of self-supervised fine-tuning for vision foundation models, where the model is adapted to a new domain without requiring annotations, leveraging only short multi-view object-centric videos. Our method is referred to as VESSA: Video-based objEct-centric Self-Supervised Adaptation for visual foundation models. VESSA's training technique is based on a self-distillation paradigm, where it is critical to carefully tune prediction heads and deploy parameter-efficient adaptation techniques - otherwise, the model may quickly forget its pretrained knowledge and reach a degraded state. VESSA benefits significantly from multi-view object observations sourced from different frames in an object-centric video, efficiently learning robustness to varied capture conditions, without the need of annotations. Through comprehensive experiments with 3 vision foundation models on 2 datasets, VESSA demonstrates consistent improvements in downstream classification tasks, compared to the base models and previous adaptation methods. Code is publicly available at https://github.com/jesimonbarreto/VESSA.

自监督学习视觉模型无监督微调视频理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。