动态融合多层视觉特征,高效实现图文对齐。
Dynamic Embedding of Hierarchical Visual Features for Efficient Vision-Language Fine-Tuning
- 根据语言模型层内表示,动态选择并融合多层视觉特征。
- 仅微调少量参数,图像问答和图像描述任务均超越现有方法。
- 适合追求高效图文对齐的模型优化场景。
大型视觉-语言模型(LVLMs)通常将视觉特征投影后与文本标记拼接,形成统一序列输入大语言模型(LLMs),导致输入序列过长,计算开销大。现有方法尝试将视觉信息融入LLM中间层,缓解序列长度问题,但常忽略模型内部的层次化语义表征及浅层视觉编码中的细粒度信息。为此,我们提出DEHVF,一种基于动态嵌入与分层视觉特征融合的高效图文微调方法。其核心利用视觉编码器与语言模型的层次表征特性,通过轻量级分层视觉融合器,依据LLM各层内部表示,动态选择对应语义粒度的分层特征,并在投影与对齐后直接嵌入对应层的前馈网络(FFN)。该方法避免序列扩展,同时动态融合多层视觉信息。仅微调少量参数,即实现跨模态信息在相同语义粒度下的精确对齐与互补。我们在ScienceQA图像问答与COCO Captions图像描述等多个基准上进行实验,结果表明,DEHVF在保持高效训练与推理的同时,精度优于现有参数高效微调(PEFT)基线。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) commonly follow a paradigm that projects visual features and then concatenates them with text tokens to form a unified sequence input for Large Language Models (LLMs). However, this paradigm leads to a significant increase in the length of the input sequence, resulting in substantial computational overhead. Existing methods attempt to fuse visual information into the intermediate layers of LLMs, which alleviate the sequence length issue but often neglect the hierarchical semantic representations within the model and the fine-grained visual information available in the shallower visual encoding layers. To address this limitation, we propose DEHVF, an efficient vision-language fine-tuning method based on dynamic embedding and fusion of hierarchical visual features. Its core lies in leveraging the inherent hierarchical representation characteristics of visual encoders and language models. Through a lightweight hierarchical visual fuser, it dynamically selects and fuses hierarchical features corresponding to semantic granularity based on the internal representations of each layer in LLMs. The fused layer-related visual features are then projected and aligned before being directly embedded into the Feed-Forward Network (FFN) of the corresponding layer in LLMs. This approach not only avoids sequence expansion but also dynamically fuses multi-layer visual information. By fine-tuning only a small number of parameters, DEHVF achieves precise alignment and complementarity of cross-modal information at the same semantic granularity. We conducted experiments across various VL benchmarks, including visual question answering on ScienceQA and image captioning on COCO Captions. The results demonstrate that DEHVF achieves higher accuracy than existing parameter-efficient fine-tuning (PEFT) baselines while maintaining efficient training and inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。