用多个专家智能体协作生成复杂文本对应的视频,提升准确性。
GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration
- 分角色智能体协同工作,分解复杂生成任务。
- 迭代验证与重设计使视频生成更符合复合文本描述。
- 自适应路由选择专用修正智能体,适配多样场景需求。
近年来,文本到视频生成模型取得了显著进展,但仍难以根据复合文本提示生成复杂动态场景,如多对象属性绑定、不同对象的时间动态性以及对象间的交互。本文提出GenMAC,一种基于多智能体协作的迭代式框架,实现复合文本到视频的生成。该框架包含设计、生成和重设计三个阶段,生成与重设计阶段形成迭代循环,逐步验证并优化生成视频。重设计阶段最具挑战性,旨在验证视频、提出修正建议,并调整文本提示、帧布局和引导尺度以进行下一轮生成。为避免单一大模型产生幻觉,将此阶段分解为四个顺序执行的基于多模态大模型(MLLM)的智能体:验证、建议、修正和输出结构化智能体。此外,针对多样化复合生成场景,设计自适应路由机制,从多个专用于特定场景的修正智能体中选择合适者。大量实验表明,GenMAC在复合文本到视频生成任务上达到当前最优性能。
原文摘要 · Abstract (English)
Text-to-video generation models have shown significant progress in the recent years. However, they still struggle with generating complex dynamic scenes based on compositional text prompts, such as attribute binding for multiple objects, temporal dynamics associated with different objects, and interactions between objects. Our key motivation is that complex tasks can be decomposed into simpler ones, each handled by a role-specialized MLLM agent. Multiple agents can collaborate together to achieve collective intelligence for complex goals. We propose GenMAC, an iterative, multi-agent framework that enables compositional text-to-video generation. The collaborative workflow includes three stages: Design, Generation, and Redesign, with an iterative loop between the Generation and Redesign stages to progressively verify and refine the generated videos. The Redesign stage is the most challenging stage that aims to verify the generated videos, suggest corrections, and redesign the text prompts, frame-wise layouts, and guidance scales for the next iteration of generation. To avoid hallucination of a single MLLM agent, we decompose this stage to four sequentially-executed MLLM-based agents: verification agent, suggestion agent, correction agent, and output structuring agent. Furthermore, to tackle diverse scenarios of compositional text-to-video generation, we design a self-routing mechanism to adaptively select the proper correction agent from a collection of correction agents each specialized for one scenario. Extensive experiments demonstrate the effectiveness of GenMAC, achieving state-of-the art performance in compositional text-to-video generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。