用小模型代理大模型，低成本训练视觉编码器并直接移植
Zero-Shot Vision Encoder Grafting via LLM Surrogates
- 用大模型浅层构建小代理模型，共享语义空间
- 移植后在多个基准上媲美全量训练效果，节省45%成本
- 适合想低成本训练视觉编码器的研究者
视觉语言模型（VLM）通常将中等规模的视觉编码器与大型语言模型（如 Llama-70B）结合，导致解码器成为训练中的主要计算负担。为降低训练成本，我们提出一种新策略：先用小型语言模型训练视觉编码器，再将其迁移至大型模型。为此，我们构建了与目标大模型共享嵌入空间和表示语言的小型‘代理模型’，通过直接继承大模型的浅层网络实现。在代理模型上训练的视觉编码器可直接迁移至大型模型中，这一过程称为零样本嫁接。实验表明，嫁接后的模型性能超过原代理对，在部分基准上甚至达到全量解码器训练的效果。使用 Llama-70B 作为解码器时，该方法整体训练成本降低约 45%。代码已开源。
原文摘要 · Abstract (English)
Vision language models (VLMs) typically pair a modestly sized vision encoder with a large language model (LLM), e.g., Llama-70B, making the decoder the primary computational burden during training. To reduce costs, a potential promising strategy is to first train the vision encoder using a small language model before transferring it to the large one. We construct small "surrogate models" that share the same embedding space and representation language as the large target LLM by directly inheriting its shallow layers. Vision encoders trained on the surrogate can then be directly transferred to the larger model, a process we call zero-shot grafting -- when plugged directly into the full-size target LLM, the grafted pair surpasses the encoder-surrogate pair and, on some benchmarks, even performs on par with full decoder training with the target LLM. Furthermore, our surrogate training approach reduces overall VLM training costs by ~45% when using Llama-70B as the decoder. The code is at https://github.com/facebookresearch/zero.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。