让小模型学会大模型的视觉理解能力，跨架构知识迁移新方法
GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
- 通过特征重校准机制，实现不同架构大模型到小模型的知识迁移
- 在多个基准上显著提升小模型性能，超越部分开源与闭源大模型
- 适用于各类视觉语言模型，无需针对特定模型设计
近期视觉语言模型（VLM）借助大语言模型（LLM）实现了与GPT-4V等闭源系统相当的性能。然而，由于计算开销巨大，在资源受限设备上部署仍具挑战。这促使研究者将大VLM知识蒸馏至更高效的小模型。当前主要挑战在于VLM架构差异：不同模型基于各异的LLM，使用不同的词元类型，包括词汇量、分词方式和词元索引顺序的差异。为突破对特定VLM类型的限制，本文提出通用型蒸馏框架GenRecal。该框架引入重校准器（Recalibrator），对异构VLM间的特征表示进行对齐与适配，实现跨类型有效知识迁移。在多个高难度基准上的广泛实验表明，GenRecal显著提升基线性能，最终超越大规模开源与闭源VLM。
原文摘要 · Abstract (English)
Recent advancements in vision-language models (VLMs) have leveraged large language models (LLMs) to achieve performance on par with closed-source systems like GPT-4V. However, deploying these models in real-world scenarios, particularly on resource-constrained devices, remains challenging due to their substantial computational demands. This has spurred interest in distilling knowledge from large VLMs into smaller, more efficient counterparts. A key challenge arises here from the diversity of VLM architectures, which are built on different LLMs and employ varying token types-differing in vocabulary size, token splits, and token index ordering. To address this challenge of limitation to a specific VLM type, we present Generation after Recalibration (GenRecal), a general-purpose distillation framework for VLMs. GenRecal incorporates a Recalibrator that aligns and adapts feature representations between heterogeneous VLMs, enabling effective knowledge transfer across different types of VLMs. Through extensive experiments on multiple challenging benchmarks, we demonstrate that GenRecal significantly improves baseline performances, eventually outperforming large-scale open- and closed-source VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。