通过分步对齐与轻量特化,让视觉语言模型用更少令牌实现高效推理。
Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Models
- 先用低秩投影压缩多层视觉特征,再加轻量Transformer捕捉结构
- 仅用144个视觉令牌,在多个评测中表现优于传统方法
- 适合需要高效推理的部署场景,尤其对算力有限的设备友好
多模态大语言模型融合视觉与文本表征以实现丰富推理能力,但处理高分辨率图像时密集视觉令牌带来的高计算成本仍是主要瓶颈。核心环节是视觉投影器,其将视觉编码器输出映射到语言模型空间。标准设计常采用简单多层感知机进行直接映射,但该方式在高分辨率输入下扩展性差,引入显著冗余。本文提出Delta-LLaVA,一种高效的投影器,采用低秩DeltaProjection将多层级视觉特征对齐至紧凑子空间后进一步交互。在此基础对齐之上,添加轻量Transformer块作为特化层,可在有限令牌预算下捕捉全局与局部结构。大量实验与消融分析表明,这种‘基线对齐-特化’设计在多个基准测试中均实现稳定提升,且仅需144个视觉令牌。该设计使推理吞吐量最高提升55%,预训练阶段训练速度加快近4-5倍,微调阶段提升超1.5倍,凸显了其在效率与可扩展性上的双重优势。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) combine visual and textual representations to enable rich reasoning capabilities. However, the high computational cost of processing dense visual tokens remains a major bottleneck. A critical component in this pipeline is the visual projector, which bridges the vision encoder and the language model. Standard designs often employ a simple multi-layer perceptron for direct token mapping, but this approach scales poorly with high-resolution inputs, introducing significant redundancy. We present Delta-LLaVA, a token-efficient projector that employs a low-rank DeltaProjection to align multi-level vision features into a compact subspace before further interaction. On top of this base alignment, lightweight Transformer blocks act as specialization layers, capturing both global and local structure under constrained token budgets. Extensive experiments and ablations demonstrate that this base-then-specialize design yields consistent gains across multiple benchmarks with only 144 tokens, highlighting the importance of token formation prior to scaling interaction capacity. With Delta-LLaVA, inference throughput improves by up to 55%, while end-to-end training accelerates by nearly 4-5x in pretraining and over 1.5x in finetuning, highlighting the dual benefits of our design in both efficiency and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。