提出解耦式数据配方方法,让视觉语言模型训练更可复现、可优化。
DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

- 将数据配比拆分为跨能力与同类别两部分,分别用搜索和优化求解。
- 在小规模代理上找到的最优配比可直接扩展到大规模训练,无需重调。
- 用800亿额外多模态token训练的模型,性能媲美更大规模开源模型。
尽管视觉语言模型(VLM)的数据筛选日益活跃,但公开的预训练数据混合构建仍主要依赖经验:从业者堆叠通过质量过滤的数据集,凭直觉设定跨领域比例,缺乏可解释的新增数据准入标准,且前沿配方未公开。本文将数据构建建模为系统性混合优化问题,通过解耦为两个正交子问题——跨能力类别间比例与类别内比例——将其转化为可复现的工程范式。对于跨类别分配,采用单变量迭代搜索;对于类别内组合,基于数据集级的质量与难度评分,建立带多样性目标的约束凸优化模型。DecoupleMix框架具备两大核心能力:指导下一步应收集哪些数据,使数据验证成为可控、可归因的实验。实验表明,该方法持续优于启发式基线。更重要的是,小规模代理上发现的最优比例可无缝迁移至大规模训练,无需重新调整。使用800亿额外多模态继续预训练令牌后,所训练的VLM在性能上可与使用显著更大多模态预算的强开源模型相媲美。
原文摘要 · Abstract (English)
While data curation for Vision Language Models (VLMs) is increasingly active, public practice for constructing pretraining mixtures remains largely heuristic: practitioners stack datasets that pass quality filters, set cross-domain ratios by intuition, and lack a principled, attributable criterion for admitting new data, while frontier recipes remain undisclosed. We formulate data construction as a systematic mixture-optimization problem and turn it into a reproducible engineering discipline by decoupling the mixture into two orthogonal sub-problems: inter-class ratios across capabilities and intra-class ratios within a category. For inter-class allocation, we use a single-variable iterative search; for intra-class composition, we apply a multidimensional, dataset-level assessment scoring Quality and Difficulty, and formulate selection as a constrained convex optimization with a diversity objective. The DecoupleMix framework delivers two critical capabilities: guiding what data to collect next and rendering dataset validation a controlled, attributable experiment. Experiments show our approach consistently surpasses heuristic baselines. Moreover, optimal ratios discovered on small-scale proxies transfer seamlessly to larger scales without retuning. Using 80B additional multimodal continue-pretraining tokens, our VLM is competitive with strong open-source models trained with substantially larger multimodal budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。