arXiv:2607.26596cs.CVcs.AI2026-07

用轻量模块替换视觉层,高效适配多模态大模型。

Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution

  • 将视觉与文本路径分离,仅微调一个轻量视觉变换器。
  • 在LLaVA-1.5上仅训练少量参数,仍达主流性能水平。
  • 适合资源受限场景下的多模态模型快速优化。

多模态大语言模型(MLLM)通过统一的Transformer架构融合视觉与文本理解,但全参数微调成本高昂且常不必要,因深层网络中视觉与文本表征需求差异显著。本文提出解耦视觉处理(DVP)框架:将预训练语言模型的上半部分解码器替换为独立可训练的轻量级单变压器块,专用于视觉令牌处理。在前半段共享处理后,视觉与文本令牌分流——视觉令牌经新初始化的单变压器块,文本令牌沿原冻结解码器继续。两路输出拼接后输入语言建模头。训练时仅更新该单变压器块,大幅减少可训练参数。在LLaVA-1.5框架上的实验表明,DVP在MME、POPE和ChartQA基准上表现优异,仅需极少参数即可达成竞争性结果,证明视觉表征可通过解耦、高效的路径有效学习。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture. However, fine-tuning all parameters of these models for visual instruction tuning is computationally expensive and often unnecessary, as the representation requirements for visual and textual tokens diverge significantly in the deeper layers of the network. In this paper, we propose Decoupled Visual Processing (DVP), an efficient training framework that replaces the upper decoder layers of a pretrained LLM with a lightweight, independently trainable single transformer block dedicated exclusively to visual token processing. Specifically, after shared processing through the first half of the decoder layers, visual and textual tokens are split: visual tokens are routed through a newly initialized single transformer block while textual tokens continue through the original frozen decoder layers. The two streams are then concatenated before the language modeling head. During training, only the single transformer block is updated, dramatically reducing the number of trainable parameters. Experiments on the LLaVA-1.5 framework demonstrate that DVP achieves competitive performance on MME, POPE, and ChartQA benchmarks while training only a fraction of the total parameters, suggesting that visual representations in MLLMs can be effectively learned through a decoupled, parameter-efficient pathway.

多模态参数效率视觉处理Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。