用语言嵌入组合解释图像，让视觉模型更透明且可复用。
LangBridge: Interpreting Image as a Combination of Language Embeddings
- 将图像嵌入显式映射为语言模型词向量的线性组合。
- 在多个大模型间直接迁移，性能几乎无损失。
- 无需重新训练，适合快速适配不同语言模型。
近年来，大型视觉-语言模型（LVLM）在复杂跨模态任务中达到人类水平表现。主流方法沿用LLaVA范式，通过两阶段训练（跨模态对齐预训练+指令微调）使用浅层MLP实现视觉-语言对齐，但其内在机制仍不清晰。尽管已有研究分析语言模型如何处理视觉标记，但对基础对齐机制的研究仍不足，且MLP适配器需更换语言模型时重新训练。为此，我们首先探究了MLP适配器的工作原理，发现其逐步将视觉嵌入投影到对应文本嵌入张成的子空间中。基于此，提出LangBridge：一种显式将视觉标记映射为语言模型词汇嵌入线性组合的新适配器。该设计支持预训练免重训的跨模型迁移，在保持性能的同时实现可解释性对齐。实验表明，基于Qwen2-0.5B预训练的LangBridge适配器可直接应用于更大模型如LLaMA3-8B或Qwen2.5-14B，仍保持竞争力。总体而言，LangBridge通过将视觉表示锚定于语言模型词嵌入，实现了可解释的对齐，并具备即插即用特性，显著提升多模型复用效率。
原文摘要 · Abstract (English)
Recent years have witnessed remarkable advances in Large Vision-Language Models (LVLMs), which have achieved human-level performance across various complex vision-language tasks. Following LLaVA's paradigm, mainstream LVLMs typically employ a shallow MLP for visual-language alignment through a two-stage training process: pretraining for cross-modal alignment followed by instruction tuning. While this approach has proven effective, the underlying mechanisms of how MLPs bridge the modality gap remain poorly understood. Although some research has explored how LLMs process transformed visual tokens, few studies have investigated the fundamental alignment mechanism. Furthermore, the MLP adapter requires retraining whenever switching LLM backbones. To address these limitations, we first investigate the working principles of MLP adapters and discover that they learn to project visual embeddings into subspaces spanned by corresponding text embeddings progressively. Based on this insight, we propose LangBridge, a novel adapter that explicitly maps visual tokens to linear combinations of LLM vocabulary embeddings. This innovative design enables pretraining-free adapter transfer across different LLMs while maintaining performance. Our experimental results demonstrate that a LangBridge adapter pre-trained on Qwen2-0.5B can be directly applied to larger models such as LLaMA3-8B or Qwen2.5-14B while maintaining competitive performance. Overall, LangBridge enables interpretable vision-language alignment by grounding visual representations in LLM vocab embedding, while its plug-and-play design ensures efficient reuse across multiple LLMs with nearly no performance degradation. See our project page at https://curryx-001.github.io/LangBridge.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。