无需标签数据，联合优化视觉Transformer量化，实现低比特高效部署。
Joint Post-Training Quantization of Vision Transformers with Learned Prompt-Guided Data Generation
- 全模型层与块间依赖联合优化，单卡一小时完成量化。
- 在ViT-small上实现W4A4和W3A3最优精度，首次达成W1.58A8高保真度。
- 用学习提示生成多样化无标签图像，性能媲美真实数据校准。
我们提出一种面向ImageNet图像分类任务的Vision Transformer端到端联合后训练量化框架。与以往基于标签数据或分块重建的方法不同，该方法在无标签条件下联合优化所有层及块间依赖关系，可有效扩展至大规模样本，且仅需单张GPU运行一小时即可完成ViT-small的量化。在ImageNet上取得了当前最优的W4A4和W3A3精度；据我们所知，首次在极低比特设置（如W1.58A8）下保持ViT、DeiT和Swin-T模型的强准确性，展现了边缘部署的巨大潜力。此外，引入一种无需数据的校准策略：利用由学习得到的多模态提示引导Stable Diffusion Turbo生成多样化无标签样本。通过增强提示嵌入与生成图像特征的多样性，该方法性能媲美真实ImageNet数据校准，显著优于简单文本提示基线（如"a <adjective> photo of <adjective> <cls>"）。
原文摘要 · Abstract (English)
We present a framework for end-to-end joint quantization of Vision Transformers trained on ImageNet for the purpose of image classification. Unlike prior post-training or block-wise reconstruction methods, we jointly optimize over the entire set of all layers and inter-block dependencies without any labeled data, scaling effectively with the number of samples and completing in just one hour on a single GPU for ViT-small. We achieve state-of-the-art W4A4 and W3A3 accuracies on ImageNet and, to the best of our knowledge, the first PTQ results that maintain strong accuracy on ViT, DeiT, and Swin-T models under extremely low-bit settings (W1.58A8), demonstrating the potential for efficient edge deployment. Furthermore, we introduce a data-free calibration strategy that synthesizes diverse, label-free samples using Stable Diffusion Turbo guided by learned multi-mode prompts. By encouraging diversity in both the learned prompt embeddings and the generated image features, our data-free approach achieves performance on par with real-data ImageNet calibration and surpasses simple text-prompt baselines such as "a <adjective> photo of <adjective> <cls>".
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。