用原型生成合成数据,让多模态小数据集训练更高效。
Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis
- 用CLIP提取图文嵌入,通过原型引导生成新数据。
- 在极小数据集下仍保持高性能,跨架构泛化能力强。
- 无需训练优化,适合资源有限的多模态研究者。
多模态学习虽在视觉-语言任务中取得显著进展,但依赖大规模图像-文本数据集,导致训练成本高、效率低。现有数据筛选与剪枝方法仍需较大子集维持性能,且在极小子集下表现不佳。数据蒸馏是潜在解决方案,但现有方法需全量数据训练并联合优化图像像素与文本特征,具有架构依赖性,限制跨架构泛化。为此,我们提出一种无学习的数据蒸馏框架,避免大规模训练与优化,提升跨架构通用性。该方法利用CLIP提取对齐的图文嵌入,获取原型,并通过unCLIP解码器合成图像,实现高效可扩展的多模态数据蒸馏。大量实验表明,本方法持续优于基于优化的数据蒸馏与子集选择方法,在跨架构泛化上达到当前最佳效果。
原文摘要 · Abstract (English)
Recent advances in multimodal learning have achieved remarkable success across diverse vision-language tasks. However, such progress heavily relies on large-scale image-text datasets, making training costly and inefficient. Prior efforts in dataset filtering and pruning attempt to mitigate this issue, but still require relatively large subsets to maintain performance and fail under very small subsets. Dataset distillation offers a promising alternative, yet existing multimodal dataset distillation methods require full-dataset training and joint optimization of image pixels and text features, making them architecture-dependent and limiting cross-architecture generalization. To overcome this, we propose a learning-free dataset distillation framework that eliminates the need for large-scale training and optimization while enhancing generalization across architectures. Our method uses CLIP to extract aligned image-text embeddings, obtains prototypes, and employs an unCLIP decoder to synthesize images, enabling efficient and scalable multimodal dataset distillation. Extensive experiments demonstrate that our approach consistently outperforms optimization-based dataset distillation and subset selection methods, achieving state-of-the-art cross-architecture generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。