通过捕捉令牌动态交互,实现多模态大模型高效压缩。
Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions
- 基于视觉-指令与生成内交互机制设计新蒸馏框架
- 蒸馏后模型性能超越更大原版模型,提升7.0%
- 适合追求轻量化且保持强跨模态能力的部署场景
多模态大语言模型(MLLMs)展现出强大的跨模态能力,但其庞大参数量带来显著部署挑战。知识蒸馏(KD)是压缩模型的可行方案,但现有方法多依赖静态的下一个词对齐,忽视了动态令牌交互所蕴含的关键理解与生成能力。为此,我们提出Align-TI框架,从令牌交互角度出发。该方法基于两个核心交互:视觉-指令间交互用于提取相关视觉信息,响应内部交互用于保证生成连贯性。Align-TI引入两个组件:IVA使学生模型模仿教师在显著视觉区域上的信息提取能力;TPA通过对序列令牌间转移概率对齐,捕捉教师动态生成逻辑。大量实验表明,Align-TI表现优越,相比基础蒸馏提升2.6%相对性能,其精简版Align-TI-2B甚至超过LLaVA-1.5-7B(更大模型)7.0%,建立新的蒸馏性能基准。代码已公开于https://github.com/lchen1019/Align-TI。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) demonstrate impressive cross-modal capabilities, yet their substantial size poses significant deployment challenges. Knowledge distillation (KD) is a promising solution for compressing these models, but existing methods primarily rely on static next-token alignment, neglecting the dynamic token interactions, which embed essential capabilities for multimodal understanding and generation. To this end, we introduce Align-TI, a novel KD framework designed from the perspective of Token Interactions. Our approach is motivated by the insight that MLLMs rely on two primary interactions: vision-instruction token interactions to extract relevant visual information, and intra-response token interactions for coherent generation. Accordingly, Align-TI introduces two components: IVA enables the student model to imitate the teacher's instruction-relevant visual information extract capability by aligning on salient visual regions. TPA captures the teacher's dynamic generative logic by aligning the sequential token-to-token transition probabilities. Extensive experiments demonstrate Align-TI's superiority. Notably, our approach achieves $2.6\%$ relative improvement over Vanilla KD, and our distilled Align-TI-2B even outperforms LLaVA-1.5-7B (a much larger MLLM) by $7.0\%$, establishing a new state-of-the-art distillation framework for training parameter-efficient MLLMs. Code is available at https://github.com/lchen1019/Align-TI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。