自动构建视觉语言模型训练数据混合，省时高效。
MaD-Mix: Multi-Modal Data Mixtures via Latent Space Coupling for Vision-Language Model Training
- 通过隐空间耦合实现跨模态对齐，自动生成数据混合比例。
- 图像文本指令微调中减少22%训练步数，效果媲美人工调参。
- 支持缺失模态数据，适合大规模多模态模型训练场景。
视觉语言模型通常在多种多模态领域上训练，但现有方法依赖昂贵的手动调参。本文提出MaD-Mix，一种原理清晰且计算高效的框架，用于生成视觉语言模型的多模态数据混合。MaD-Mix将数据混合建模为模态感知的域对齐最大化问题，通过模态间耦合变量获得闭式多模态对齐分数。该方法系统性处理缺失模态的领域，支持纯文本数据的整合。在0.5B和7B模型上的实证评估表明，MaD-Mix显著加速了多种基准上的模型训练。在图像-文本指令微调中，仅用22%更少的训练步骤即达到与人工调参相当的效果。在复杂的三模态视频-图像-文本场景中，当人工调参变得不切实际时，MaD-Mix以可忽略的混合计算开销（<1 GPU小时）提升平均准确率，支持现代视觉语言模型流水线的可扩展混合设计。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are typically trained on a diverse set of multi-modal domains, yet current practices rely on costly manual tuning. We propose MaD-Mix, a principled and computationally efficient framework that derives multi-modal data mixtures for VLM training. MaD-Mix formulates data mixing as modality-aware domain alignment maximization and obtains closed-form multi-modal alignment scores from the Fenchel dual through inter-modal coupling variables. MaD-Mix systematically handles domains with missing modalities, allowing for the integration of language-only domains. Empirical evaluations across 0.5B and 7B models demonstrate that MaD-Mix accelerates VLM training across diverse benchmarks. MaD-Mix matches human-tuned data mixtures using 22% fewer training steps in image-text instruction tuning. In complex tri-modal video-image-text scenarios, where manual tuning becomes impractical, MaD-Mix boosts average accuracy over uniform weights, with negligible mixture computation overhead (< 1 GPU-hour), enabling scalable mixture design for modern VLM pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。