提出轻量级视觉编码器融合框架LEO,提升多模态大模型视觉理解能力。
Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs
- 采用独立投影+动态分块融合,实现多视觉编码器的高效整合
- 在11个基准上多数任务超越现有MoVE方法,高分辨率输入表现优异
- 无需修改架构即可适配自动驾驶等专业领域,通用性强
混合视觉编码器(MoVE)已成为增强多模态大语言模型(MLLMs)细粒度视觉理解能力的重要方法,显著提升其在复杂光学字符识别和场景理解等任务中的表现。然而,如何有效融合不同编码器及其视觉标记,并支持高分辨率输入,仍是开放挑战。本文系统研究了基于MoVE的融合设计,提出了在标记层面整合互补编码器的原则。研究发现,一种包含后适应融合、独立投影器、块级序列交错和带全局上下文的动态分块的轻量级方案,在多个基准上均表现强劲。据此构建的LEO架构简单高效,在11个视觉-语言基准上多数任务优于现有MoVE方法。此外,LEO无需调整架构或训练流程即可有效适配自动驾驶等专用领域,性能媲美主流基线,展现出良好泛化能力。代码已开源。
原文摘要 · Abstract (English)
Mixture of Vision Encoders (MoVE) has emerged as a powerful approach to enhance the fine-grained visual understanding of multimodal large language models (MLLMs), improving their ability to handle tasks such as complex optical character recognition and scene understanding. Despite these advances, effectively combining diverse encoders and their visual tokens, while also scaling to high-resolution inputs, remains an open challenge. In this work, we conduct a systematic study of fusion designs for MoVE-based MLLMs, highlighting principles for token-level integration across complementary encoders. Our study shows that a lightweight recipe consisting of post-adaptation fusion with independent projectors, tile-level sequence interleaving, and dynamic tiling with global context delivers strong performance on diverse benchmarks. We integrate these principles into a simple and effective architecture that we call LEO. Extensive evaluation on 11 vision-language benchmarks demonstrates that LEO achieves better results on the majority of tasks compared to existing MoVE-based approaches. Furthermore, LEO adapts effectively to the specialized domain of autonomous driving without altering its architecture or training recipe, achieving competitive performance against established baselines and thereby highlighting its ability to generalize. The code is available at https://github.com/Mozhgan91/LEO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。