arXiv:2601.21426cs.CV2026-01

用大模型生成伪图文对,让分类模型在微调时也能用多模态能力。

MultiModal Fine-tuning with Synthetic Captions

  • 用MLLM为单模态图像生成高质量合成描述,实现多模态微调。
  • 在13个数据集上提升分类性能,少样本场景下效果更显著。
  • 适合希望利用预训练多模态模型的视觉任务研究者。

本文针对深度神经网络预训练与微调之间的根本性差距:尽管预训练已从单模态转向多模态并提升了视觉理解能力,但微调仍以单模态为主,限制了丰富预训练表征的潜力。为此,我们提出一种新方法,利用多模态大语言模型(MLLM)为微调数据生成合成图像描述,将单模态数据集转化为多模态数据集。通过精心设计的提示,结合类别标签和领域上下文,生成适用于分类任务的高质量描述。此外,我们引入监督对比损失函数,显式鼓励同类别表征聚类,并提出一种新推理技术,利用每张图像多个合成描述的类别平均文本嵌入。在13个图像分类基准上的大量实验表明,该方法优于基线,尤其在少样本学习场景中表现突出。本工作建立了一种新的数据增强范式,有效弥合了多模态预训练与微调间的鸿沟。代码已公开于https://github.com/s-enmt/MMFT。

原文摘要 · Abstract (English)

In this paper, we address a fundamental gap between pre-training and fine-tuning of deep neural networks: while pre-training has shifted from unimodal to multimodal learning with enhanced visual understanding, fine-tuning predominantly remains unimodal, limiting the benefits of rich pre-trained representations. To bridge this gap, we propose a novel approach that transforms unimodal datasets into multimodal ones using Multimodal Large Language Models (MLLMs) to generate synthetic image captions for fine-tuning models with a multimodal objective. Our method employs carefully designed prompts incorporating class labels and domain context to produce high-quality captions tailored for classification tasks. Furthermore, we introduce a supervised contrastive loss function that explicitly encourages clustering of same-class representations during fine-tuning, along with a new inference technique that leverages class-averaged text embeddings from multiple synthetic captions per image. Extensive experiments across 13 image classification benchmarks demonstrate that our approach outperforms baseline methods, with particularly significant improvements in few-shot learning scenarios. Our work establishes a new paradigm for dataset enhancement that effectively bridges the gap between multimodal pre-training and fine-tuning. Our code is available at https://github.com/s-enmt/MMFT.

多模态微调合成数据少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。