arXiv:2511.15411cs.CVcs.LG2025-11中稿 · CVPR

无需真实数据,让CLIP模型量化后仍保持高精度。

D4C: Data-Free Quantization for Contrastive Language-Image Pre-training Models

  • 用文本提示和对比生成,合成语义丰富且结构多样的伪图像。
  • 在4位量化下,比现有方法性能提升超过10%(如在ImageNet上)。
  • 适合隐私敏感场景下的视觉语言模型压缩,尤其适用于CLIP类模型。

数据无关量化(DFQ)为模型压缩提供了实用方案,无需访问真实数据,特别适合隐私敏感场景。尽管DFQ在单模态模型中表现良好,但其在对比语言-图像预训练(CLIP)等多模态模型上的应用仍不充分。本文揭示,直接将现有DFQ技术应用于CLIP会导致显著性能下降,原因在于合成样本语义信息不足、图像内部多样性低。为此,我们提出D4C——首个专为CLIP设计的DFQ框架。D4C通过三个关键组件合成高质量伪图像:1)提示引导语义注入,利用文本提示对齐真实语义;2)结构对比生成,通过前景-背景对比合成复现自然图像的组合结构;3)扰动感知增强,在可控扰动下提升样本多样性和鲁棒性。三者协同使合成图像兼具语义信息与结构多样性,有效弥合了CLIP在DFQ下的性能差距。大量实验验证了D4C的有效性,在多种位宽和模型上均取得显著提升。

原文摘要 · Abstract (English)

Data-Free Quantization (DFQ) offers a practical solution for model compression without requiring access to real data, making it particularly attractive in privacy-sensitive scenarios. While DFQ has shown promise for unimodal models, its extension to Vision-Language Models such as Contrastive Language-Image Pre-training (CLIP) models remains underexplored. In this work, we reveal that directly applying existing DFQ techniques to CLIP results in substantial performance degradation due to two key limitations: insufficient semantic content and low intra-image diversity in synthesized samples. To tackle these challenges, we propose D4C, the first DFQ framework tailored for CLIP. D4C synthesizes semantically rich and structurally diverse pseudo images through three key components: 1) Prompt-Guided Semantic Injection aligns generated images with real-world semantics using text prompts; 2) Structural Contrastive Generation reproduces compositional structures of natural images by leveraging foreground-background contrastive synthesis; and 3) Perturbation-Aware Enhancement applies controlled perturbations to improve sample diversity and robustness. These components jointly empower D4C to synthesize images that are both semantically informative and structurally diverse, effectively bridging the performance gap of DFQ on CLIP. Extensive experiments validate the effectiveness of D4C, showing significant performance improvements on various bit-widths and models.

模型压缩量化CLIP无数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。