不微调模型,用压缩嵌入拼接实现跨模态分类新突破
Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach

- 各模态输入经冻结模型提取后,用PCA压缩并拼接输入表格基础模型
- 在超过50万样本、2000类的复杂任务上零微调达顶尖性能
- 适合追求快速部署且无需训练的新任务场景
我们提出CoMET(Composing Modality Encoders with Tabular foundation models),一种无需微调的简单而高效的多模态分类方法:将每种模态通过冻结的预训练主干网络处理,用PCA压缩得到的嵌入,再拼接输入到表格式基础模型(TFM)进行预测。实验表明,仅靠PCA即可作为有效适配器,在多种模态上均表现强劲且鲁棒。当基础模型的[CLS]标记与下游任务对齐不佳时,我们引入轻量级自适应池化器PALPooling,持续提升表示质量。通过组合强大的冻结表示学习主干与TFM,该方法在多个多样化多模态基准上实现无训练的顶尖性能。在具有大量细粒度类别的层级任务中,方法支持快速可扩展分类,可在不微调的情况下处理超50万样本、2000类的数据集。结果表明,基础模型的组合是一种简单但强大的开箱即用方案,挑战了针对新问题必须依赖复杂端到端训练流程的必要性。
原文摘要 · Abstract (English)
We introduce CoMET, \textit{\textbf{C}omposing \textbf{M}odality \textbf{E}ncoders with \textbf{T}abular foundation models}, a simple yet highly competitive method for multimodal classification: pass each modality through a frozen pre-trained backbone, compress the resulting embeddings with PCA, and concatenate as input into a Tabular Foundation Model (TFM) for prediction. We show that PCA alone suffices to act as an adaptor yielding strong, robust performance across modalities. When the \texttt{CLS} tokens of the foundation model align poorly with downstream tasks, we propose \textbf{PALPooling}, a lightweight adaptive token pooler that consistently improves representation quality. By composing strong frozen representation learning backbones with TFMs, our approach achieves state-of-the-art results across diverse multimodal benchmarks without any training. On hierarchical tasks with large fine-grained class spaces, our approach enables fast and scalable classification, handling datasets with over 500,000 samples and 2,000 classes without any fine-tuning. Overall, our results show that the composition of foundation models is a simple, yet powerful, out-of-the-box solution for multimodal learning, challenging the necessity of complex, end-to-end training pipelines for new problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。