用多个视觉编码器动态选择，提升特定领域图文模型性能。
MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processing
- 根据输入内容自动选最优视觉编码器，如Unichat、InternViT。
- 在ChartQA等多任务上表现优于单一编码器，无需图像分块处理。
- 适合需要跨领域精准理解的图文应用，如图表问答。
多模态语言模型通过适配器将视觉编码器与大语言模型结合。现有方法通常依赖单一预训练视觉编码器，而不同专业编码器在特定领域可显著提升性能。本文提出MOVE（视觉编码器混合）方法，通过自动路由机制，将输入分配给最合适的候选编码器（如Unichat、InternViT、Texify），从而在ChartQA、MMBench、MMMU等多个基准上实现优异表现。实验表明，该方法在不引入高分辨率图像切片复杂性的情况下，仍能获得具有竞争力的准确率。
原文摘要 · Abstract (English)
Multimodal language models (MLMs) integrate visual and textual information by coupling a vision encoder with a large language model through the specific adapter. While existing approaches commonly rely on a single pre-trained vision encoder, there is a great variability of specialized encoders that can boost model's performance in distinct domains. In this work, we propose MOVE (Mixture of Vision Encoders) a simple yet effective approach to leverage multiple pre-trained encoders for specialized multimodal tasks. MOVE automatically routes inputs to the most appropriate encoder among candidates such as Unichat, InternViT, and Texify, thereby enhancing performance across a diverse set of benchmarks, including ChartQA, MMBench, and MMMU. Experimental results demonstrate that MOVE achieves competitive accuracy without incurring the complexities of image slicing for high-resolution images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。