用掩码扩散模型让图像自动生成描述，学出强视觉特征
Masked Diffusion Captioning for Visual Feature Learning
- 用图像条件掩码扩散语言模型生成文本描述
- 在多个数据集上性能媲美自回归与对比学习方法
- 适合想用生成式训练学视觉特征的研究者
我们通过图像条件掩码扩散语言模型对图像进行描述生成来学习视觉特征，这一方法称为掩码扩散描述（Masked Diffusion Captioning, MDC）。训练时，图像-描述对中的文本标记以随机比例被掩码，基于视觉特征的解码器被训练以重建原始文本。训练完成后，学习到的视觉特征可应用于下游视觉任务。与自回归描述不同，MDC中视觉学习信号不依赖于文本序列中每个词的位置，从而减少了对辅助目标的需求。在线性探测实验中，多种学术规模模型和数据集上的结果表明，该方法学到的视觉特征在性能上可与自回归及对比学习方法相媲美。
原文摘要 · Abstract (English)
We learn visual features by captioning images with an image-conditioned masked diffusion language model, a formulation we call masked diffusion captioning (MDC). During training, text tokens in each image-caption pair are masked at a randomly chosen ratio, and a decoder conditioned on visual features is trained to reconstruct the original text. After training, the learned visual features can be applied to downstream vision tasks. Unlike autoregressive captioning, the strength of the visual learning signal in MDC does not depend on each token's position in the sequence, reducing the need for auxiliary objectives. Linear probing experiments across a variety of academic-scale models and datasets show that the learned visual features are competitive with those produced by autoregressive and contrastive approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。