用多智能体协作让大模型学会看懂图像,轻量高效还开源。
Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaboration
- 分角色协作:视觉感知用小模型,逻辑推理用大模型,各司其职。
- 轻量模型超越大厂闭源系统,在多项任务上胜过GPT-4o。
- 模块化设计,可灵活扩展到新模态和新领域,适合研究者复现与部署。
大型语言模型(LLM)在知识密集型推理任务中表现卓越,但将其扩展至新模态(如视觉)通常需构建大规模视觉语言模型(VLM),成本高昂。小型VLM虽高效灵活,却缺乏前沿LLM的广博知识与推理能力。本文提出BeMyEyes,一种模块化多智能体框架,通过对话协调高效的感知型VLM与强大的推理型LLM,实现跨模态推理。我们设计数据合成与监督微调流程,训练感知智能体与推理智能体协同工作。该框架无需训练大规模多模态模型,保留了LLM的泛化与推理能力,并支持灵活扩展至新领域与模态。实验表明,将文本仅模型DeepSeek-R1与Qwen2.5-VL-7B感知器结合,即可在多项知识密集型多模态任务上超越大型闭源模型GPT-4o,验证了方法的有效性、模块化与可扩展性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities in challenging, knowledge-intensive reasoning tasks. However, extending LLMs to perceive and reason over a new modality (e.g., vision), often requires costly development of large-scale vision language models (VLMs) with LLMs as backbones. Smaller VLMs are more efficient and adaptable but often lack the broad knowledge and reasoning capabilities of frontier LLMs. In this work, we propose BeMyEyes, a modular, multi-agent framework for extending LLMs to multimodal reasoning by orchestrating collaboration between efficient, adaptable VLMs as perceivers and powerful LLMs as reasoners through conversations. We then introduce a data synthesis and supervised fine-tuning pipeline to train the perceiver agent to effectively collaborate with the reasoner agent. By combining the complementary strengths of perception and reasoning agents, BeMyEyes avoids the need for training large-scale multimodal models, preserves the generalization and reasoning capabilities of LLMs, and allows flexible extension to new domains and modalities. Experiments show that our framework unlocks the multimodal reasoning capabilities for LLMs, enabling a lightweight and fully open-source solution, i.e. equipping text-only DeepSeek-R1 with Qwen2.5-VL-7B perceiver, to outperform large-scale proprietary VLMs such as GPT-4o on a wide range of knowledge-intensive multimodal tasks. These results demonstrate the effectiveness, modularity, and scalability of our multi-agent approach for building future multimodal reasoning systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。