提出高效视觉编码新方法,显著降低高分辨率图像处理的计算开销。
LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?

- 采用分片编码替代全局编码,更好保留局部细节
- 在ViT浅层引入早期压缩,降低55.8%计算量
- 适合需要高效处理高清图像的多模态模型研究
视觉编码是多模态大模型处理高分辨率图像时的主要计算瓶颈。现有方法通常先进行全局编码生成大量视觉标记,再通过后置的ViT压缩降低序列长度,但此过程仍需承担完整ViT的二次注意力计算成本。本工作从编码策略与视觉标记压缩两个维度重新审视这一范式:首先,控制实验表明分片编码在多个基准测试中优于全局编码,说明局部视图对细粒度感知更有利;其次,提出一种ViT内部早期压缩机制,在浅层即减少标记数量,显著降低视觉编码所需浮点运算量。将该压缩机制整合进分片编码框架,构建了针对高分辨率输入优化的LLaVA-UHD v4。在文档理解、OCR和通用VQA等多样任务上,其视觉编码计算量减少55.8%,同时性能持平甚至超越基线。结果表明,可在不牺牲下游任务表现的前提下大幅提升编码效率,为高效高分辨率多模态模型提供实用设计方向。所有模型权重与代码将公开发布以支持后续研究。
原文摘要 · Abstract (English)
Visual encoding constitutes a major computational bottleneck in Multimodal Large Language Models (MLLMs), especially for high-resolution image inputs. The prevailing practice typically adopts global encoding followed by post-ViT compression. Global encoding produces massive token sequences, while post-ViT compression incurs the full quadratic attention cost of the ViT before any token reduction takes place. In this work, we revisit this convention along two dimensions: the encoding strategy and visual token compression. First, controlled experiments show that slice-based encoding outperforms global encoding across benchmarks, suggesting that preserving local details through sliced views can be more beneficial than applying global attention for fine-grained perception. Second, we introduce intra-ViT early compression, which reduces tokens in shallow ViT layers and substantially lowers visual-encoding FLOPs while preserving downstream performance. By integrating intra-ViT compression into the slice-based encoding framework, we present LLaVA-UHD v4, an efficient and compute-controllable visual encoding scheme tailored for high-resolution inputs. Across a diverse set of benchmarks covering document understanding, OCR, and general VQA, LLaVA-UHD v4 reduces visual-encoding FLOPs by 55.8% while matching or even surpassing baseline performance. These results suggest that visual-encoding efficiency can be substantially improved without sacrificing downstream performance, providing a practical design direction for efficient high-resolution MLLMs. All model weights and code will be publicly released to support further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。