用纯图像数据高效预训练视觉生成模型,突破配对数据瓶颈
Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-training
- 分两阶段训练:先纯图像无监督预训练,再少量图文对微调
- 仅用1050小时算力,3.6B模型在GenEval达0.89,超越多个基线
- 适合追求高效、高质量图像生成的开发者和研究者
统一多模态模型(UMMs)的视觉生成组件常受限于低效的预训练范式和稀缺的高质量图文配对数据。本文系统分析了UMM视觉生成的预训练方案,识别出两大瓶颈。为此提出图像独占训练(IOMM),一种数据高效的两阶段训练框架:第一阶段仅使用大量未标注图像数据预训练视觉生成模块,彻底摆脱对配对数据的依赖;第二阶段融合未标注图像与少量精选图文对进行微调,提升指令对齐与生成质量。实验表明,IOMM显著提升训练效率并达到最优性能。例如,我们的IOMM-B(3.6B)模型从零训练仅需约1050小时H800 GPU时(其中1000小时用于高效图像独占预训练),在GenEval上达0.89,在WISE上达0.55,超越BAGEL-7B(0.82 & 0.55)和BLIP3-o-4B(0.84 & 0.50)。代码已公开。
原文摘要 · Abstract (English)
Unified Multimodal Models (UMMs) are often constrained by the pre-training of their $\textbf{visual generation components}$, which typically relies on inefficient paradigms and scarce, high-quality text-image paired data. In this paper, we systematically analyze pre-training recipes for $\textbf{UMM visual generation}$ and identify these two issues as the major bottlenecks. To address them, we propose $\textbf{Image-Only Training for UMMs (IOMM)}$, a data-efficient two-stage training framework. The first stage pre-trains the visual generative component $\textbf{exclusively}$ using abundant unlabeled image-only data, thereby removing the dependency on paired data $\textbf{for this costly phase}$. The second stage fine-tunes the model using a mixture of unlabeled images and a small curated set of text-image pairs, leading to improved instruction alignment and generative quality. Extensive experiments show that IOMM not only improves training efficiency but also achieves state-of-the-art (SOTA) performance. For example, our IOMM-B (3.6B) model was trained from scratch using only $\sim \textbf{1050}$ H800 GPU hours (with the vast majority, $\textbf{1000}$ hours, dedicated to the efficient $\textbf{image-only pre-training stage}$). It achieves $\textbf{0.89}$ on GenEval and $\textbf{0.55}$ on WISE--surpassing strong baselines such as BAGEL-7B (0.82 & 0.55) and BLIP3-o-4B (0.84 & 0.50). Code is available $\href{https://github.com/LINs-lab/IOMM}{https://github.com/LINs-lab/IOMM}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。