将视觉语言模型权重压缩至三值，实现高效低资源部署。
TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge
- 将双编码器权重转为三值，仅用1.58比特存储。
- 压缩比达16.98倍，推理速度提升2.3倍，内存减少10倍。
- 支持零样本图像分类与图文检索，适合边缘设备部署。
近年来，图像-文本对比建模受到广泛关注，代表性模型如对比语言-图像预训练（CLIP）。本文提出TernaryCLIP，一种轻量级计算框架，将CLIP中视觉与文本编码器的连接权重转换为三值格式，而非全精度浮点数。TernaryCLIP融合量化感知训练与知识蒸馏模块，有效防止精度下降，实现低成本、高效率计算。大量实验表明，TernaryCLIP可实现高达99%的权重三值化，仅需1.58比特表示，压缩比达16.98倍，推理加速2.3倍，存储减少16倍，内存优化10倍，同时达到60%稀疏度，在41个常用数据集上保持出色的零样本图像分类与图文检索性能。本工作验证了极端量化在大型多模态模型中的可行性，支持其在资源受限设备上的高效部署。模型与代码可在Hugging Face和GitHub获取。
原文摘要 · Abstract (English)
Recent years have witnessed an increasing interest in image-text contrastive modeling, exemplified by models such as Contrastive Language-Image Pretraining (CLIP). In this paper, we propose the TernaryCLIP, a lightweight computational framework that converts connection weights of both vision and text encoders of CLIP into the ternary format, instead of full-precision or floating ones. TernaryCLIP incorporates quantization-aware training and distillation modules, preventing precision degradation and enabling low-cost and high-efficiency computations. Comprehensive experiments demonstrate that TernaryCLIP can achieve up to 99\% ternarized weights with 1.58-bit representation, 16.98 $\times$ compression ratio, 2.3 $\times$ inference acceleration, 16 $\times$ storage reduction, 10 $\times$ memory optimization, and 60\% sparsity while maintaining promising performance on zero-shot image classification and image-text retrieval tasks across 41 commonly used datasets. Our work highlights the feasibility of extreme quantization for large multimodal models, supporting effective and efficient deployment on resource-constrained devices. The model and code can be accessed from Hugging Face and GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。