arXiv:2605.08985cs.CV2026-05

提出高效视觉编码新方法,显著降低高分辨率图像处理的计算开销。

LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?

论文配图:LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?
图 1 · 摘自论文原文
  • 采用分片编码替代全局编码,更好保留局部细节
  • 在ViT浅层引入早期压缩,降低55.8%计算量
  • 适合需要高效处理高清图像的多模态模型研究

视觉编码是多模态大模型处理高分辨率图像时的主要计算瓶颈。现有方法通常先进行全局编码生成大量视觉标记,再通过后置的ViT压缩降低序列长度,但此过程仍需承担完整ViT的二次注意力计算成本。本工作从编码策略与视觉标记压缩两个维度重新审视这一范式:首先,控制实验表明分片编码在多个基准测试中优于全局编码,说明局部视图对细粒度感知更有利;其次,提出一种ViT内部早期压缩机制,在浅层即减少标记数量,显著降低视觉编码所需浮点运算量。将该压缩机制整合进分片编码框架,构建了针对高分辨率输入优化的LLaVA-UHD v4。在文档理解、OCR和通用VQA等多样任务上,其视觉编码计算量减少55.8%,同时性能持平甚至超越基线。结果表明,可在不牺牲下游任务表现的前提下大幅提升编码效率,为高效高分辨率多模态模型提供实用设计方向。所有模型权重与代码将公开发布以支持后续研究。

原文摘要 · Abstract (English)

Visual encoding constitutes a major computational bottleneck in Multimodal Large Language Models (MLLMs), especially for high-resolution image inputs. The prevailing practice typically adopts global encoding followed by post-ViT compression. Global encoding produces massive token sequences, while post-ViT compression incurs the full quadratic attention cost of the ViT before any token reduction takes place. In this work, we revisit this convention along two dimensions: the encoding strategy and visual token compression. First, controlled experiments show that slice-based encoding outperforms global encoding across benchmarks, suggesting that preserving local details through sliced views can be more beneficial than applying global attention for fine-grained perception. Second, we introduce intra-ViT early compression, which reduces tokens in shallow ViT layers and substantially lowers visual-encoding FLOPs while preserving downstream performance. By integrating intra-ViT compression into the slice-based encoding framework, we present LLaVA-UHD v4, an efficient and compute-controllable visual encoding scheme tailored for high-resolution inputs. Across a diverse set of benchmarks covering document understanding, OCR, and general VQA, LLaVA-UHD v4 reduces visual-encoding FLOPs by 55.8% while matching or even surpassing baseline performance. These results suggest that visual-encoding efficiency can be substantially improved without sacrificing downstream performance, providing a practical design direction for efficient high-resolution MLLMs. All model weights and code will be publicly released to support further research.

多模态模型视觉编码高效计算高分辨率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。