将预训练的CLIP转为稀疏专家模型，大幅降低训练成本并提升性能。
CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling
- 用稀疏专家结构改造预训练稠密CLIP模型，实现高效训练。
- 稀疏版CLIP B/16在COCO和Flickr30k上分别领先稠密版7.2%和6.6%。
- 仅用30%推理计算量，超越更大规模的CLIP L/14模型，适合资源受限场景。
Mixture-of-Experts (MoE) 模型在控制推理成本的同时提升了模型容量。尽管将 MoE 引入 CLIP 等多模态模型能提高性能，但训练过程仍具挑战且代价高昂。我们提出一种名为 CLIP-Upcycling (CLIP-UP) 的高效训练策略，可将预训练的稠密 CLIP 模型转换为稀疏的 MoE 架构。通过大量实验验证不同设置与辅助损失，我们证明了 CLIP-UP 显著降低了训练复杂度和成本。令人瞩目的是，使用 CLIP-UP 训练的稀疏版 CLIP B/16 模型，在 COCO 与 Flickr30k 文本到图像 Recall@1 任务中分别优于其稠密版本 7.2% 和 6.6%。该模型甚至在性能上超过更大的 CLIP L/14 模型，同时仅消耗 30% 的推理浮点运算量（FLOPs）。我们进一步展示了该训练方法在不同规模下的泛化能力，确立了稀疏再利用作为一种构建高效、高性能 CLIP 模型的实用且可扩展的路径。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models are crucial for scaling model capacity while controlling inference costs. While integrating MoE into multimodal models like CLIP improves performance, training these models is notoriously challenging and expensive. We propose CLIP-Upcycling (CLIP-UP), an efficient alternative training strategy that converts a pre-trained dense CLIP model into a sparse MoE architecture. Through extensive experimentation with various settings and auxiliary losses, we demonstrate that CLIP-UP significantly reduces training complexity and cost. Remarkably, our sparse CLIP B/16 model, trained with CLIP-UP, outperforms its dense counterpart by 7.2% and 6.6% on COCO and Flickr30k text-to-image Recall@1 benchmarks respectively. It even surpasses the larger CLIP L/14 model on this task while using only 30% of the inference FLOPs. We further demonstrate the generalizability of our training recipe across different scales, establishing sparse upcycling as a practical and scalable approach for building efficient, high-performance CLIP models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。