视觉基础模型迁移效果差,因预训练目标与下游任务不匹配。
Understanding the Transfer Limits of Vision Foundation Models
- 用特征分布差异衡量预训练与任务的契合度
- 契合度高的模型提升更明显,收敛更快
- 临床影像任务验证了预训练目标设计的重要性
视觉基础模型(VFMs)虽通过大规模预训练获取广泛知识,但在下游视觉任务中表现不均,即使投入大量算力。本文认为问题源于预训练目标与下游任务需求不匹配:如掩码图像重建或对比学习,主要捕捉通用视觉模式或全局语义,难以满足分割、分类等具体任务要求。为验证这一假设,研究在五个前列腺多模态MRI任务中评估了基于MAE的重建型模型ProFound和基于对比学习的ProViCNet,分析其从预训练到微调的迁移性能。结果表明,预训练特征与下游任务间的一致性(以微调前后特征的MMD值衡量)越高,性能提升越显著,收敛越快,强调应从下游应用角度设计和评估预训练目标。
原文摘要 · Abstract (English)
Foundation models leverage large-scale pretraining to capture extensive knowledge, demonstrating generalization in a wide range of language tasks. By comparison, vision foundation models (VFMs) often exhibit uneven improvements across downstream tasks, despite substantial computational investment. We postulate that this limitation arises from a mismatch between pretraining objectives and the demands of downstream vision-and-imaging tasks. Pretraining strategies like masked image reconstruction or contrastive learning shape representations for tasks such as recovery of generic visual patterns or global semantic structures, which may not align with the task-specific requirements of downstream applications including segmentation, classification, or image synthesis. To investigate this in a concrete real-world clinical area, we assess two VFMs, a reconstruction-focused MAE-based model (ProFound) and a contrastive-learning-based model (ProViCNet), on five prostate multiparametric MR imaging tasks, examining how such task alignment influences transfer performance, i.e., from pretraining to fine-tuning. Our findings indicate that better alignment between pretraining and downstream tasks, measured by simple divergence metrics such as maximum-mean-discrepancy (MMD) between the same features before and after fine-tuning, correlates with greater performance improvements and faster convergence, emphasizing the importance of designing and analyzing pretraining objectives with downstream applicability in mind.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。