提出可训练文本编码器的多模态数据蒸馏方法，提升图文对齐效果。
CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder
- 通过跨协方差对齐真实与合成特征，联合优化图像和文本编码器
- 仅用500对合成数据，在Flickr30K和COCO上检索准确率提升6.8%
- 适合需要高效训练视觉语言模型的研究者使用
多模态数据蒸馏旨在生成少量图像-文本对，以高效训练大规模视觉-语言模型。尽管数据蒸馏在单模态任务中表现良好，但将其扩展至多模态对比学习面临关键挑战：学习跨模态对齐并应对大型编码器的高计算成本。先前方法通过冻结文本编码器、仅更新图像编码器和文本投影层来提升可扩展性，但我们发现这严重限制了语义对齐，成为性能提升的瓶颈。本文提出CovMatch，一种可扩展的数据蒸馏框架，通过对齐真实与合成特征的跨协方差，并正则化各模态内的特征分布。与之前方法不同，CovMatch支持双编码器的联合优化，实现更强的跨模态对齐和更优性能。在Flickr30K和COCO数据集上的评估表明，CovMatch优于当前最优的多模态蒸馏方法，仅使用500个合成样本即实现高达6.8%的检索准确率提升。
原文摘要 · Abstract (English)
Multimodal dataset distillation aims to synthesize a small set of image-text pairs that enables efficient training of large-scale vision-language models. While dataset distillation has shown promise in unimodal tasks, extending it to multimodal contrastive learning presents key challenges: learning cross-modal alignment and managing the high computational cost of large encoders. Prior approaches address scalability by freezing the text encoder and update only the image encoder and text projection layer. However, we find this severely limits semantic alignment and becomes a bottleneck for performance scaling. We propose CovMatch, a scalable dataset distillation framework that aligns the cross-covariance of real and synthetic features while regularizing feature distributions within each modality. Unlike prior approaches, CovMatch enables joint optimization of both encoders, leading to stronger cross-modal alignment and improved performance. Evaluated on Flickr30K and COCO, CovMatch outperforms state-of-the-art multimodal distillation methods and achieves up to 6.8% absolute gains in retrieval accuracy using only 500 synthetic pairs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。