arXiv:2603.02667cs.CVcs.LG2026-03

统一图文对比与生成任务,用动态掩码让模型同时学会对齐和生成。

Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation

  • 通过渐进式掩码调度,让同一模型在不同遮蔽比例下协同训练。
  • 仅需12.5%图像解码即可选最优生成路径,提升效率与质量。
  • 适合需要高效图文理解与生成的多任务应用开发者。

将文本-图像对比学习与文本到图像(T2I)生成统一于单一端到端模型极具挑战,因二者要求相反的掩码策略:对比对齐需几乎完整可见的视觉标记,而生成建模则需重度破坏。本文提出DREAM框架,采用掩码预热机制,使掩码分布随训练逐步迁移,使得低与高掩码率在每一步共存。这种共同暴露使单个联合训练编码器可服务于双重目标。由此带来的稳定优化实现了推理阶段语义对齐解码:文本编码器在所有掩码比例下训练,能对部分生成图像打分,并以仅12.5%图像解码量选出最佳生成轨迹,显著提升FID与吞吐量。DREAM优于单独目标基线模型CLIP与FLUID:在ImageNet线性探测(+1.1%)、5样本迁移(+4.1%)、ADE20K分割(+1.9%)和NYU深度估计(+6.25%)上超越CLIP;在CC12M FID上优于FLUID(+6.2%),同时保持CLIP Score。这些成果表明,当合理统一时,图文对比与生成目标具有协同效应而非相互竞争。

原文摘要 · Abstract (English)

Unifying text-image contrastive learning and text-to-image (T2I) generation in a single end-to-end model is challenging because the two objectives demand opposing masking regimes: contrastive alignment needs near-complete visible tokens, while masked generative modeling needs heavy corruption. We introduce DREAM, a unified framework that resolves this conflict through Masking Warmup, a schedule that shifts the center of the masking distribution over training, so low and high masking ratios coexist at every step. This co-exposure lets a single jointly-trained encoder serve both objectives. The resulting stable optimization unlocks Semantically Aligned Decoding at inference: the text encoder, trained against visual embeddings at all masking ratios, can score partially generated images and select the best trajectory with as little as 12.5% of the image decoded, improving both FID and throughput. DREAM outperforms its single-objective baselines, CLIP and FLUID: on ImageNet linear-probing (+1.1%), 5-shot transfer (+4.1%), ADE20K segmentation (+1.9%), and NYU depth estimation (+6.25%) over CLIP, and on CC12M FID (+6.2%) over FLUID while maintaining CLIP Score. Together, these gains show that text-image contrastive and generative objectives, when properly unified, are synergistic rather than competing.

图文对齐生成模型联合训练高效生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。