用自适应运输融合多模型,提升零样本分类效果
SOTA: Self-adaptive Optimal Transport for Zero-Shot Classification with Multiple Foundation Models
- 不训练地自适应融合多个基础模型输出
- 跨领域测试均显著优于单个模型
- 适合需要快速部署零样本分类的场景
基础模型因其强大的零样本分类能力受到广泛关注。本文基于两个关键观察:(1) 视觉-语言模型(如CLIP)过度依赖类别文本先验,难以捕捉细粒度视觉线索;而纯视觉基础模型(如DINO)提供丰富且区分性强的视觉特征,但缺乏语义对齐;(2) 不同视觉-语言模型在不同数据集上表现差异大,源于预训练差异。为此,我们提出SOTA(自适应最优传输),一种无需训练的集成框架,通过学习自适应传输计划融合多个基础模型(视觉-语言或纯视觉)的输出。该方法无需先验知识,能自动平衡各模型贡献。在自然图像、医学病理和遥感等多样领域广泛实验验证了SOTA的泛化能力,结果一致表明其有效利用不同模型的互补优势,并显著超越单个模型。代码已开源。
原文摘要 · Abstract (English)
Foundation models have attracted widespread attention across domains due to their powerful zero-shot classification capabilities. This work is motivated by two key observations: (1) \textit{Vision-Language Models} (VLMs), such as CLIP, often over-rely on class-level textual priors and struggle to capture fine-grained visual cues, whereas \textit{Vision-only Foundation Models} (VFMs), such as DINO, provide rich and discriminative visual features but lack semantic alignment; (2) the performance of different VLMs varies considerably across datasets owing to differences in pre-training. To address these challenges, we propose \textbf{SOTA} (\textit{Self-adaptive Optimal TrAnsport}), a \textit{training-free} ensemble framework that integrates the outputs of multiple foundation models~(VFMs or VLMs) by learning a self-adaptive transport plan. Notably, \textbf{SOTA} is prior-free and automatically balances model contributions. Extensive experiments across diverse domains, including natural images, medical pathology, and remote sensing, validate the generalizability of \textbf{SOTA}. The results consistently show that it effectively leverages the complementary strengths of different foundation models and achieves substantial improvements over individual models. The implementation code is available at: https://github.com/Afleve/self-adaptive-Optimal-Transport.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。