用语言化层间对齐，让小模型高效复现大模型的视觉推理能力。
VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models
- 通过逐层引入语言化中间映射，实现大小模型间的渐进式知识迁移。
- 2B和7B模型在10个基准上分别提升11.0%和17.4%，超越GPT-4V表现。
- 适合移动端、机器人等资源受限场景下的高效视觉语言模型部署。
闭源视觉语言模型（如GPT-4V）产生的高质量视觉指令数据推动了开源VLM在多种规模下的快速发展。然而，使用更大模型提升性能会带来显著计算开销，尤其在移动平台和机器人等资源受限设备上难以部署。为此，我们提出VLsI：一种2B和7B参数量的新型VLM家族，注重效率且不牺牲准确率。VLsI采用独特的逐层蒸馏机制，引入中间“语言化器”（verbalizers），将每层特征映射到自然语言空间，使小型VLM能灵活对齐大型VLM的推理过程。该方法缓解了输出模仿中的训练不稳定性问题，并超越传统仅对齐最后一层的做法，实现小模型与大模型在各层进展上的对齐。我们在十个挑战性视觉语言基准上验证了VLsI，无需模型缩放、合并或架构修改，2B和7B模型分别相比GPT-4V提升11.0%和17.4%。
原文摘要 · Abstract (English)
The recent surge in high-quality visual instruction tuning samples from closed-source vision-language models (VLMs) such as GPT-4V has accelerated the release of open-source VLMs across various model sizes. However, scaling VLMs to improve performance using larger models brings significant computational challenges, especially for deployment on resource-constrained devices like mobile platforms and robots. To address this, we propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which prioritizes efficiency without compromising accuracy. VLsI leverages a unique, layer-wise distillation process, introducing intermediate "verbalizers" that map features from each layer to natural language space, allowing smaller VLMs to flexibly align with the reasoning processes of larger VLMs. This approach mitigates the training instability often encountered in output imitation and goes beyond typical final-layer tuning by aligning the small VLMs' layer-wise progression with that of the large ones. We validate VLsI across ten challenging vision-language benchmarks, achieving notable performance gains (11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling, merging, or architectural changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。