arXiv:2503.21307cs.CVcs.AI2025-03被引 27

通过三种压缩方法提升视觉编码效率,实现更快更强的多模态模型。

InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compression

  • 引入PVTC、LVTC和RVTC三种视觉令牌压缩技术,优化视觉特征处理流程。
  • 仅用20%以下视觉令牌即达顶尖性能,12项任务平均指标提升2.34%。
  • 适合追求高效多模态推理与部署的研究者及工程团队。

多数多模态大语言模型将视觉令牌视为文本序列,将其与文本令牌一同输入大语言模型(LLM)。然而大量视觉令牌显著增加计算资源与时间需求。本文提出InternVL-X,通过引入三种视觉令牌压缩方法,在性能与效率上均超越原InternVL模型。首先,提出新型视觉-语言投影器PVTC,将邻近视觉嵌入整合为局部查询,并利用变换后的CLS token作为全局查询,通过局部与全局查询执行点对区域交叉注意力,更高效地转换视觉特征。其次,设计逐层视觉令牌压缩模块LVTC,于浅层压缩令牌,深层通过上采样与残差连接恢复,大幅提高计算效率。此外,提出高效高分辨率切片方法RVTC,根据图像面积或长度动态调整视觉令牌数量,仅轻微损失性能即可显著提升训练效率。使用20%或更少的视觉令牌,InternVL-X在7个公开多模态大模型基准上达到当前最优表现,12项任务平均指标提升2.34%。

原文摘要 · Abstract (English)

Most multimodal large language models (MLLMs) treat visual tokens as "a sequence of text", integrating them with text tokens into a large language model (LLM). However, a great quantity of visual tokens significantly increases the demand for computational resources and time. In this paper, we propose InternVL-X, which outperforms the InternVL model in both performance and efficiency by incorporating three visual token compression methods. First, we propose a novel vision-language projector, PVTC. This component integrates adjacent visual embeddings to form a local query and utilizes the transformed CLS token as a global query, then performs point-to-region cross-attention through these local and global queries to more effectively convert visual features. Second, we present a layer-wise visual token compression module, LVTC, which compresses tokens in the LLM shallow layers and then expands them through upsampling and residual connections in the deeper layers. This significantly enhances the model computational efficiency. Futhermore, we propose an efficient high resolution slicing method, RVTC, which dynamically adjusts the number of visual tokens based on image area or length filtering. RVTC greatly enhances training efficiency with only a slight reduction in performance. By utilizing 20% or fewer visual tokens, InternVL-X achieves state-of-the-art performance on 7 public MLLM benchmarks, and improves the average metric by 2.34% across 12 tasks.

多模态视觉压缩高效推理大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。