arXiv:2604.14198cs.LGcs.AI2026-04被引 1

通过双轴优化数据混合,提升多模态大模型中段训练效率与泛化能力。

MixAtlas: Uncertainty-aware Data Mixture Optimization for Multimodal LLM Midtraining

  • 将数据按视觉概念与任务类型双维度分解,构建可搜索的混合空间
  • 在0.5B小模型上搜索,使7B模型性能平均提升8.5%-17.6%
  • 发现的训练配方可跨模型规模迁移,节省训练步数最多达2倍

领域重加权能提升样本效率和下游泛化能力,但多模态中段训练中的数据混合优化仍缺乏探索。现有方法仅沿单一维度(如数据格式或任务类型)调整混合比例。本文提出MixAtlas,一种生成可解释、可迁移的基准目标数据配方的方法。该方法将训练语料库沿两个维度分解:图像概念(基于CLIP嵌入发现的10个视觉领域聚类)与任务监督(包括图像描述、OCR、定位、检测和VQA等5种目标类型)。使用小规模代理模型(Qwen2-0.5B)结合高斯过程代理与GP-UCB采样策略,在相同代理预算下,比基于回归的基线更高效地找到性能更优的混合方案。在涵盖视觉理解、文档推理和多模态推理的10个基准上评估:Qwen2-7B模型性能平均提升8.5%-17.6%;Qwen2.5-7B提升1.0%-3.3%;两种设置均能在最多减少两倍训练步数的情况下达到基线损失水平。0.5B代理发现的配方可成功迁移至7B级训练,覆盖多个Qwen模型系列。

原文摘要 · Abstract (English)

Domain reweighting can improve sample efficiency and downstream generalization, but data-mixture optimization for multimodal midtraining remains largely unexplored. Current multimodal training recipes tune mixtures along a single dimension, typically data format or task type. We introduce MixAtlas, a method that produces benchmark-targeted data recipes that can be inspected, adapted, and transferred to new corpora. MixAtlas decomposes the training corpus along two axes: image concepts (10 visual-domain clusters discovered via CLIP embeddings) and task supervision (5 objective types including captioning, OCR, grounding, detection, and VQA). Using small proxy models (Qwen2-0.5B) paired with a Gaussian-process surrogate and GP-UCB acquisition, MixAtlas searches the resulting mixture space with the same proxy budget as regression-based baselines but finds better-performing mixtures. We evaluate on 10 benchmarks spanning visual understanding, document reasoning, and multimodal reasoning. On Qwen2-7B, optimized mixtures improve average performance by 8.5%-17.6% over the strongest baseline; on Qwen2.5-7B, gains are 1.0%-3.3%. Both settings reach baseline-equivalent training loss in up to 2 times fewer steps. Recipes discovered on 0.5B proxies transfer to 7B-scale training across Qwen model families.

多模态数据混合训练优化迁移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。