用多智能体框架统一图文音视频理解与生成,无需联合训练
A Unified Multi-Agent Framework for Universal Multimodal Understanding and Generation
- 分认知与推敲两阶段,通过角色化智能体协作完成多模态理解与生成
- 在MME基准上超越GPT-4o,在图像、视频、音频生成中均表现优异
- 支持任意模态间转换,模块化设计便于扩展,适合多模态系统开发者
现实世界中的多模态应用常需实现任意模态间的双向能力,涵盖文本、图像、音频和视频的理解与生成。然而,将自回归语言模型(LLMs)的推理优势与扩散模型的高保真生成能力融合仍具挑战。现有方法依赖僵化流程或紧密耦合架构,限制灵活性与可扩展性。本文提出MAGUS(Multi-Agent Guided Unified Multimodal System),一种模块化框架,通过解耦的两个阶段——认知与推敲——统一多模态理解与生成。认知阶段由三个角色驱动的多模态LLM智能体(感知者、规划者、反思者)在共享文本空间中协同对话,实现结构化理解与规划。推敲阶段引入生长感知搜索机制,使基于LLM的推理与基于扩散模型的生成相互增强。MAGUS支持即插即用扩展、可扩展的任意模态转换与语义对齐,且无需联合训练。在多个基准测试中,包括图像、视频和音频生成以及跨模态指令遵循任务,MAGUS均优于强基线与前沿系统。尤其在MME基准上,其性能超过强大闭源模型GPT-4o。
原文摘要 · Abstract (English)
Real-world multimodal applications often require any-to-any capabilities, enabling both understanding and generation across modalities including text, image, audio, and video. However, integrating the strengths of autoregressive language models (LLMs) for reasoning and diffusion models for high-fidelity generation remains challenging. Existing approaches rely on rigid pipelines or tightly coupled architectures, limiting flexibility and scalability. We propose MAGUS (Multi-Agent Guided Unified Multimodal System), a modular framework that unifies multimodal understanding and generation via two decoupled phases: Cognition and Deliberation. MAGUS enables symbolic multi-agent collaboration within a shared textual workspace. In the Cognition phase, three role-conditioned multimodal LLM agents - Perceiver, Planner, and Reflector - engage in collaborative dialogue to perform structured understanding and planning. The Deliberation phase incorporates a Growth-Aware Search mechanism that orchestrates LLM-based reasoning and diffusion-based generation in a mutually reinforcing manner. MAGUS supports plug-and-play extensibility, scalable any-to-any modality conversion, and semantic alignment - all without the need for joint training. Experiments across multiple benchmarks, including image, video, and audio generation, as well as cross-modal instruction following, demonstrate that MAGUS outperforms strong baselines and state-of-the-art systems. Notably, on the MME benchmark, MAGUS surpasses the powerful closed-source model GPT-4o.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。