提出重要性感知二值化,让大模型压缩更智能、更省资源。
SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models

- 基于输入模态构建显著性图,区分单模态与跨模态激活权重
- 在约1比特压缩下,比现有方法在多个数据集上提升5%以上准确率
- 适合需要部署轻量化多模态模型的开发者和边缘计算场景
大型视觉语言模型(LVLMs)在多模态理解方面取得显著进展,但其庞大的参数量和跨模态计算带来巨大内存与延迟开销,严重限制了在资源受限设备上的实际部署。二值化通过大幅降低存储与计算成本提供了一种有吸引力的解决方案。然而,现有方法忽视了不同层与模态间权重的重要性差异,导致无关任务的参数被无谓保留,而关键模态权重又未充分优化,造成显著性能下降。为此,我们提出一种新颖的显著性感知二值化方法(SAB-LVLM)。具体而言,针对文本与视觉输入构建海森矩阵,设计空间显著性图以区分单一模态激活的全精度权重与跨模态激活的权重,并提出模态引导融合策略,生成跨层与跨模态的显著性感知二值化图。该图作为误差重加权项融入二值化目标,通过交替显著性加权更新方案实现二值化拟合。大量实验表明,在约1比特压缩约束下,SAB-LVLM优于现有二值化量化方法。代码已开源:https://github.com/LyuQi127/SAB_LVLM。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal understanding, yet their enormous parameter scale and cross-modal computation incur substantial memory and latency overhead, severely limiting real-world deployment on resource-constrained devices. Binarization offers an attractive solution by drastically reducing storage and computational costs. However, existing binarization methods neglect the varying importance of weights across different layers and modalities. This causes parameters irrelevant to downstream tasks to be unnecessarily retained, whereas modality-critical weights may not be adequately optimized, resulting in significant performance degradation. To address these challenges, we develop a novel \underline{S}ignificance-\underline{A}ware \underline{B}inarization for \underline{L}arge \underline{V}ision-\underline{L}anguage \underline{M}odels (SAB-LVLM). Specifically, after constructing Hessian matrices for textual and visual inputs, we propose a spatial significance map to distinguish full-precision weights activated under a single modality from those activated across modalities. We then devise a modality-guided integration strategy to obtain the significance-aware binarization map, which measures weight significance across layers and modalities. Subsequently, this binarization map is incorporated into the binarization objective as an error reweighting term, and binarization fitting is performed through an alternating significance-weighted update scheme. Extensive experiments illustrate the superiority of our SAB-LVLM over existing binary PTQ methods under an approximately 1-bit compression constraint. Our code is accessible at https://github.com/LyuQi127/SAB_LVLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。