提出细粒度注意力引导,让小模型更精准模仿大模型的视觉关注点。
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation

- 聚焦生成响应时的视觉注意力,而非输入提示时的注意力。
- 按每个词元的注意力熵自适应加权,提升蒸馏精度。
- 在多个基准上超越现有方法,适合多模态模型轻量化部署。
尽管知识蒸馏广泛用于通过大模型监督训练轻量级模型,但仅依赖输出词元分布对压缩多模态大语言模型(MLLM)效果有限。由于输出词元是模型关注视觉输入的产物,已有研究尝试显式蒸馏注意力以提供直接监督信号。然而,究竟应蒸馏哪些注意力信号仍不明确。本文发现:下游性能与响应-视觉注意力相似性高度相关,而与提示-视觉注意力关联极弱。此外,注意力分布存在显著词元间差异,说明统一蒸馏目标次优。为此,我们提出词元级响应-视觉注意力引导(TRAG),一方面将重点转向响应-视觉信号,另一方面基于注意力熵自适应加权KL散度,有效引导学生模型精确复现教师模型的视觉关注点。在多个基准上的实验证明,TRAG显著优于现有蒸馏基线。
原文摘要 · Abstract (English)
While knowledge distillation (KD) is widely adopted for training lightweight models by leveraging supervision from larger teacher models, relying solely on output token distributions has proven insufficient for compressing Multimodal Large Language Models (MLLMs). Since output tokens are a byproduct of the model attending to visual inputs, prior works have explored explicitly distilling attention to provide a direct supervisory signal. While promising, the precise utility of which attention signals to distill remains under-explored. In this work, we challenge the conventional reliance on prompt-to-vision attention by revealing that downstream performance correlates strongly with response-to-vision attention similarity to the teacher, but negligibly with that of prompt-conditioned attention. Furthermore, we observe that attention distributions exhibit significant variance across individual tokens, indicating that a uniform distillation objective is suboptimal. To this end, we introduce Token-level Response-visual Attention Guidance (TRAG), a distillation objective that 1) shifts the focus to response-to-vision signals and 2) employs token-specific objectives by adaptively weighting the Kullback-Leibler divergence based on attention entropy, effectively guiding the student to mirror the teacher's precise visual focus. Extensive experimental results on multiple benchmarks demonstrate that TRAG significantly outperforms prior distillation baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。