arXiv:2601.13886cs.CV2026-01被引 3

融合视觉语言与自监督学习,提升模型的全局语义与局部精度。

Revisiting Multi-Task Visual Representation Learning

  • 设计多任务框架MTV,联合优化语义对齐、自监督重建和密集空间监督。
  • 在多个基准上实现超越CLIP和MAE的性能,兼顾全局理解与细粒度定位。
  • 利用深度估计和目标检测模型生成伪标签,无需人工标注即可大规模训练。

当前视觉表征学习呈现两极分化:视觉语言模型(如CLIP)擅长全局语义对齐但缺乏空间精度,自监督方法(如MAE、DINO)能捕捉精细局部结构却难以建模高层语义。本文认为两类范式本质互补,可通过多任务框架整合,并借助密集空间监督进一步增强。提出MTV框架,通过共享主干网络联合优化视觉-语言对比、自监督重建和密集空间三项目标。为避免人工标注,利用高容量“专家”模型(如Depth Anything V2和OWLv2)大规模生成结构化伪标签。此外,系统分析了多任务学习的机制:(i) 各目标的边际增益,(ii) 任务间的协同与干扰关系,(iii) 不同数据与模型规模下的扩展行为。结果表明,MTV实现‘兼得二者之优’,显著提升细粒度空间推理能力,同时保持强全局语义理解。研究证明,基于高质量伪监督的多任务学习是构建通用视觉编码器的可扩展路径。

原文摘要 · Abstract (English)

Current visual representation learning remains bifurcated: vision-language models (e.g., CLIP) excel at global semantic alignment but lack spatial precision, while self-supervised methods (e.g., MAE, DINO) capture intricate local structures yet struggle with high-level semantic context. We argue that these paradigms are fundamentally complementary and can be integrated into a principled multi-task framework, further enhanced by dense spatial supervision. We introduce MTV, a multi-task visual pretraining framework that jointly optimizes a shared backbone across vision-language contrastive, self-supervised, and dense spatial objectives. To mitigate the need for manual annotations, we leverage high-capacity "expert" models -- such as Depth Anything V2 and OWLv2 -- to synthesize dense, structured pseudo-labels at scale. Beyond the framework, we provide a systematic investigation into the mechanics of multi-task visual learning, analyzing: (i) the marginal gain of each objective, (ii) task synergies versus interference, and (iii) scaling behavior across varying data and model scales. Our results demonstrate that MTV achieves "best-of-both-worlds" performance, significantly enhancing fine-grained spatial reasoning without compromising global semantic understanding. Our findings suggest that multi-task learning, fueled by high-quality pseudo-supervision, is a scalable path toward more general visual encoders.

多任务学习视觉表征伪标签自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。