高剪枝率下用图像描述补偿信息，提升视觉语言模型效率
Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models
- 用轻量级图文描述模型生成与问题相关的视觉内容
- 在6.5%更少浮点运算下，性能比当前最优高出20.6%
- 无需人工标注，自监督学习实现高效训练，适合资源受限场景
大型视觉语言模型（LVLM）虽表现优异，但计算开销大，主要源于输入视觉序列包含数百甚至上千个标记。现有方法通过去除冗余标记降低开销，但在高剪枝率下因视觉信息丢失导致性能严重下降。本文提出自适应内容补偿方法（ACCM），通过图像描述有效缓解信息损失。ACCM包含轻量级描述生成模型与选择器：前者在用户指令引导下生成相关描述，后者从多个候选中选出最适配的描述。借助自监督学习，模块可无需人工或自动标注即可高效训练。在七个基准上的实验表明，ACCM显著优于现有方法，在浮点运算减少6.5%时，性能超越当前最优达20.6%。
原文摘要 · Abstract (English)
Despite the great success of Large Vision Language Models (LVLMs), their high computational cost severely limits their broad applications. The computational cost of LVLMs mainly stems from the visual sequence of the input, which consists of hundreds or even thousands of tokens. Although existing methods have made progress by removing redundant tokens, they suffer from severe performance degradation with high pruning rates due to the loss of visual information. In this paper, we propose an Adaptive Content Compensation Method (ACCM), which can effectively mitigate the visual information loss via an image caption. Specifically, ACCM comprises two key components: a lightweight caption model and a selector. Firstly the caption model generates question-related descriptions under the guidance of the user instruction. Then the selector further identifies a contextually appropriate caption from multiple candidates. Leveraging self-supervised learning, our modules could be learned efficiently without any human or automated labeling. We conduct extensive experiments across seven benchmarks and the results show that ACCM significantly outperforms existing methods with lower FLOPs (e.g., surpassing SOTA by 20.6% with 6.5% fewer FLOPs).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。