将多模态数据插入模型中层，显著降低训练与推理开销。
DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding
- 多模态数据不拼接在开头，而是直接插入模型中层以跳过早期计算
- 在350M至13B参数模型上，训练和推理成本下降，性能持平或更好
- 适用于视觉、音频、分子数据等多模态任务，适合高效部署场景
大型语言模型（LLMs）的数据与参数规模持续增长，但带来的性能提升逐渐减弱，训练成本却持续攀升，亟需更高效的微调与推理方法，尤其对多模态语言模型（MLMs）而言，多模态输入处理的开销限制了其实际应用。近期研究发现，大型MLMs深层存在隐式的跨模态对齐现象，揭示了模型如何编码信息。受此启发，并基于我们观察到的多模态交互主要发生在模型深层的现象，我们提出一种简单修改：不再将多模态数据与语言提示在初始阶段拼接，而是将其直接插入模型中间层，从而完全绕过早期层。在多种模态（如视觉使用LLaVA & BLIP，音频使用LTU，分子数据使用MoLCA）和模型规模（从350M到13B参数）上的实验表明，该方法有效降低训练与推理成本，同时至少保持，甚至超越现有基线的性能表现。
原文摘要 · Abstract (English)
Hyperscaling of data and parameter count in LLMs is yielding diminishing improvement when weighed against training costs, underlining a growing need for more efficient finetuning and inference without sacrificing performance. This is especially so for multimodal language models (MLMs), where the overhead of processing multimodal tokens can limit their practical viability. Parallely, recent work has uncovered implicit cross-modal alignment in the deeper layers of large MLMs, deepening our understanding of how MLMs process and encode information. Motivated by this, and our observation that MLMs naturally defer most cross-modal token interactions to deeper layers of the model, we propose a simple modification. Instead of concatenation with the language prompt at the start, we insert multimodal tokens directly into the middle, allowing them to entirely bypass the early layers. Our results with diverse modalities, (i) LLaVA \& BLIP for vision, (ii) LTU for audio, and (iii) MoLCA for molecular data, and model sizes, starting from 350M to 13B parameters, indicate that our method reduces both training and inference costs, while at least preserving, if not surpassing the performance of existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。