提升高分辨率图像理解能力，避免切图导致的语义断裂。
FILA: Fine-Grained Vision Language Models
- 设计混合视觉编码器，融合局部与全局图像特征。
- 在TextVQA和DocVQA上分别提升9.6%和6.9%性能。
- 适合需要精准图文理解的视觉语言模型研究者。
近年来，多模态大语言模型（MLLMs）处理高分辨率图像的能力受到关注。当前常见方法是将原始高分辨率图像动态裁剪为小块子图像，输入在低分辨率图像上预训练的视觉编码器。然而，该方法常导致图像中物体与关联区域被截断，造成语义断裂。为此，我们提出HyViLM，可处理任意分辨率图像并保留整体上下文。具体而言：(i) 设计新型视觉编码器Hybrid Encoder，不仅编码子图像，还与详细全局视觉特征交互，显著提升高分辨率图像编码能力；(ii) 提出优化的特征融合策略，有效利用视觉编码器不同层的信息。在相同设置下，相较于现有先进MLLMs，HyViLM在10项任务中表现更优，其中在TextVQA任务上提升9.6%，在DocVQA任务上提升6.9%。
原文摘要 · Abstract (English)
Recently, there has been growing interest in the capability of multimodal large language models (MLLMs) to process high-resolution images. A common approach currently involves dynamically cropping the original high-resolution image into smaller sub-images, which are then fed into a vision encoder that was pre-trained on lower-resolution images. However, this cropping approach often truncates objects and connected areas in the original image, causing semantic breaks. To address this limitation, we introduce HyViLM, designed to process images of any resolution while retaining the overall context during encoding. Specifically, we: (i) Design a new visual encoder called Hybrid Encoder that not only encodes individual sub-images but also interacts with detailed global visual features, significantly improving the model's ability to encode high-resolution images. (ii) Propose an optimal feature fusion strategy for the dynamic cropping approach, effectively leveraging information from different layers of the vision encoder. Compared with the state-of-the-art MLLMs under the same setting, our HyViLM outperforms existing MLLMs in nine out of ten tasks. Specifically, HyViLM achieves a 9.6% improvement in performance on the TextVQA task and a 6.9% enhancement on the DocVQA task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。