通过视觉验证增强融合多模型,提升视觉推理效率与准确性。
Vision Verification Enhanced Fusion of VLMs for Efficient Visual Reasoning
- 结合视觉与语言模态,用焦点误差多样性筛选互补模型。
- 在四个基准上超越单个最优模型,最高提升8.09%准确率。
- 适合需要抗幻觉、高鲁棒性的复杂视觉推理任务。
随着视觉语言模型(VLMs)数量和多样性的增长,许多研究探索基于语言的集成、协作与路由技术以提升多模态推理能力。本文提出一种融合视觉与语言模态的多样化模型选择方法,引入焦点误差多样性来捕捉不同VLM间的互补推理,并设计基于核函数相关性分析(CKA-focal)的焦点多样性度量,评估其视觉特征表示的分歧程度。在候选VLM池构建的集成表面上,采用遗传算法有效剔除对融合性能无贡献的模型组件。我们识别出每项任务的最佳模型组合,并融合各模型输出,证明异构模型能动态捕获认知不确定性,缓解幻觉问题。所提出的V3Fusion方法可生成双焦点多样性融合预测,在无多数共识或多数模型错误时仍保持高性能。在四个主流VLM基准(A-OKVQA、MMMU、MMMU-Pro、OCR-VQA)上的实验表明,该方法在MMMU上比最优VLM提升8.09%,在MMMU-Pro上提升4.87%;在生成任务中,优于Intern-VL2-8b和Qwen2.5-VL-7b,在A-OKVQA和OCR-VQA上表现领先。代码与数据集已开源。
原文摘要 · Abstract (English)
With the growing number and diversity of Vision-Language Models (VLMs), many works explore language-based ensemble, collaboration, and routing techniques across multiple VLMs to improve multi-model reasoning. In contrast, we address the diverse model selection using both vision and language modalities. We introduce focal error diversity to capture complementary reasoning across VLMs and a CKA-based focal diversity metric (CKA-focal) to measure disagreement in their visual embeddings. On the constructed ensemble surface from a pool of candidate VLMs, we applied a Genetic Algorithm to effectively prune out those component VLMs that do not add value to the fusion performance. We identify the best combination for each task as well as fuse the outputs of each VLMs in the model pool, and show that heterogeneous models can capture epistemic uncertainty dynamically and mitigate hallucinations. Our V3Fusion approach is capable of producing dual focal-diversity fused predictions with high performance for vision-language reasoning, even when there is no majority consensus or the majority of VLMs make incorrect predictions. Extensive experiments validate V3Fusion on four popular VLM benchmarks (A-OKVQA, MMMU, MMMU-Pro, and OCR-VQA). The results show that V3Fusion outperforms the best-performing VLM on MMMU by 8.09% and MMMU-Pro by 4.87% gain in accuracy. For generative tasks, V3Fusion outperforms Intern-VL2-8b and Qwen2.5-VL-7b, the top-2 VLM performers on both A-OKVQA and OCR-VQA. Our code and datasets are available at https://github.com/sftekin/v3fusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。