通过对齐视觉注意力提升小模型的组合推理能力
CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs
- 提出注意力对齐的蒸馏框架，解决小模型视觉感知不足问题
- 在组合推理任务上显著提升性能，同时保持VQA表现
- 适用于多种骨干网络，通用性强，适合部署优化
近期，高效多模态大语言模型（MLLMs）因应对高计算复杂度而备受关注，使其更适用于实际应用。知识蒸馏（KD）作为一项有前景的技术，可将大型模型（教师）中的丰富视觉与语言知识迁移到小型模型（学生）中。然而，我们发现现有KD方法难以有效传递教师模型的视觉感知能力，这一挑战此前被忽视。通过系统分析，我们识别出学生与教师之间视觉注意力错位是主要原因。基于此，提出CompoDistill框架，显式对齐学生模型的视觉注意力以增强其视觉感知能力。大量实验表明，CompoDistill在需要视觉感知的组合推理任务上显著提升性能，同时保持原有视觉问答任务的良好表现。此外，该方法在更先进的骨干网络上也展现有效性，证明其通用性。
原文摘要 · Abstract (English)
Recently, efficient Multimodal Large Language Models (MLLMs) have gained significant attention as a solution to their high computational complexity, making them more practical for real-world applications. In this regard, the knowledge distillation (KD) approach has emerged as a promising alternative, which transfers the rich visual and linguistic knowledge from a larger model (teacher) to a smaller model (student). However, we observe that existing KD methods struggle to effectively distill the teacher MLLM's rich visual perception abilities to the student, a challenge that has been largely overlooked in previous studies. Through a systematic analysis, we identify visual attention misalignment between student and teacher as the main cause of this issue. Based on this insight, we propose CompoDistill, a novel KD framework that explicitly aligns the student's visual attention with that of the teacher to enhance the student's visual perception abilities. Our extensive experiments show that CompoDistill significantly improves performance on compositional reasoning tasks that require visual perception abilities while maintaining strong performance on visual question answering tasks, as done in existing studies. Furthermore, CompoDistill demonstrates effectiveness with a more advanced backbone, highlighting its generalizability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。