多视觉编码器的MLLM实际冗余严重,有时删掉反而更好。
Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders
- 通过掩码测试发现多个视觉编码器常可互换,部分任务中单个编码器主导。
- 特定任务上删减编码器能提16%准确率,整体性能提升3.6%。
- 提出两个量化指标,帮助诊断模型冗余,适合优化架构设计者。
近期多模态大语言模型(MLLMs)越来越多地集成多个视觉编码器,假设不同预训练目标能提供互补的视觉信号。然而我们发现这一假设在实践中常不成立。通过对代表性多编码器MLLM进行系统性编码器掩码实验,发现移除部分编码器通常导致性能温和下降,甚至偶尔提升,揭示了普遍存在的编码器冗余。为此,我们引入两个严谨度量:条件利用率(CUR),衡量某编码器在其他编码器存在下的边际贡献;信息差距(IG),捕捉模型内编码器效用的异质性。实验显示:(i) 在OCR和图表任务中表现出强专业化,单一编码器的CUR超过90%;(ii) 通用视觉问答和知识类任务中冗余显著,编码器基本可互换;(iii) 存在负向贡献的有害编码器(CUR为负)。值得注意的是,掩码特定编码器可在特定任务类别上实现最高16%的准确率提升,并带来3.6%的整体性能增益。此外,单编码器或双编码器变体在多数非OCR任务上恢复超过90%基线性能,且训练资源与推理延迟显著降低。该分析挑战了‘越多编码器越好’的直觉,为构建更高效、有效的多模态架构提供了可操作的诊断工具。
原文摘要 · Abstract (English)
Recent multimodal large language models (MLLMs) increasingly integrate multiple vision encoders to improve performance on various benchmarks, assuming that diverse pretraining objectives yield complementary visual signals. However, we show this assumption often fails in practice. Through systematic encoder masking across representative multi encoder MLLMs, we find that performance typically degrades gracefully, and sometimes even improves, when selected encoders are masked, revealing pervasive encoder redundancy. To quantify this effect, we introduce two principled metrics: the Conditional Utilization Rate (CUR), which measures an encoder s marginal contribution in the presence of others, and the Information Gap (IG), which captures heterogeneity in encoder utility within a model. Using these tools, we observe: (i) strong specialization on tasks like OCR and Chart, where a single encoder can dominate with a CUR greater than 90 percent, (ii) high redundancy on general VQA and knowledge based tasks, where encoders are largely interchangeable, (iii) instances of detrimental encoders with negative CUR. Notably, masking specific encoders can yield up to 16 percent higher accuracy on a specific task category and 3.6 percent overall performance boost compared to the full model.Furthermore, single and dual encoder variants recover over 90 percent of baseline on most non OCR tasks with substantially lower training resources and inference latency. Our analysis challenges the more encoders are better heuristic in MLLMs and provides actionable diagnostics for developing more efficient and effective multimodal architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。