用信心门控蒸馏提升视觉语言模型量化性能，兼顾精度与效率。
Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
- 基于信息瓶颈原理，融合蒸馏与量化感知训练，约束信息容量同时保留关键知识。
- INT4模型在SQA和MMBench上超越FP16基线，最高达76.9分，接近教师模型性能。
- 适合资源受限场景部署，实测吞吐提升3倍，内存减少54%。
视觉语言模型（VLMs）虽具备强大多模态能力，但部署成本高，后训练量化常导致显著精度下降。尽管潜力巨大，面向VLM的量化感知训练仍研究不足。本文提出GRACE框架，将知识蒸馏与量化感知训练统一于信息瓶颈原则下：量化限制信息容量，蒸馏则指导在该预算内保留何种信息。将教师模型视为任务相关知识的代理，引入信心门控解耦蒸馏以过滤不可靠监督，采用关系中心核对齐传递视觉标记结构，并通过拉格朗日松弛设计自适应控制器，平衡保真度与容量约束。在LLaVA与Qwen系列模型的广泛基准测试中，我们的INT4模型持续优于FP16基线（如LLaVA-1.5-7B在SQA上为70.1 vs. 66.8；Qwen2-VL-2B在MMBench上为76.9 vs. 72.6），几乎达到教师模型性能。使用真实INT4内核，实现3倍吞吐提升与54%内存降低。该原则性框架显著优于现有量化方法，为资源受限部署提供了有力解决方案。代码与数据见：https://github.com/ForeverBlue816/GRACE。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss. Despite its potential, quantization-aware training for VLMs remains underexplored. We propose GRACE, a framework unifying knowledge distillation and QAT under the Information Bottleneck principle: quantization constrains information capacity while distillation guides what to preserve within this budget. Treating the teacher as a proxy for task-relevant information, we introduce confidence-gated decoupled distillation to filter unreliable supervision, relational centered kernel alignment to transfer visual token structures, and an adaptive controller via Lagrangian relaxation to balance fidelity against capacity constraints. Across extensive benchmarks on LLaVA and Qwen families, our INT4 models consistently outperform FP16 baselines (e.g., LLaVA-1.5-7B: 70.1 vs. 66.8 on SQA; Qwen2-VL-2B: 76.9 vs. 72.6 on MMBench), nearly matching teacher performance. Using real INT4 kernel, we achieve 3$\times$ throughput with 54% memory reduction. This principled framework significantly outperforms existing quantization methods, making GRACE a compelling solution for resource-constrained deployment. Code and data are available at: https://github.com/ForeverBlue816/GRACE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。