用多个AI代理分任务纠错,让复杂图像生成更准。
Marmot: Object-Level Self-Correction via Multi-Agent Reasoning
- 拆解任务到每个物体,分别修正属性与位置
- 多轮修正不累积错误,生成图像更准确
- 适合需要精确控制物体数量和关系的场景
尽管扩散模型在生成高质量图像方面表现优异,但在复杂多物体场景中,常出现计数不准、属性错误及空间关系错乱的问题。现有方法依赖多模态大语言模型作为AI代理构建自纠正框架,但往往过度依赖模型能力,难以覆盖所有物体,且多轮编辑易导致误差累积。为此,我们提出Marmot——一种基于多智能体推理的多物体自纠正通用框架,以提升图像-文本对齐效果。首先,利用大语言模型作为感知对象的智能体,根据图像描述自动将自纠正任务分解为以物体为中心的子任务。针对每个子任务,构建包含决策-执行-验证机制的对象修正系统,仅作用于单个物体的分割掩码或物体对的边界框,有效降低物体间干扰,提升编辑可靠性。为高效融合子任务结果并避免多阶段编辑带来的误差累积,提出像素域拼接平滑器,采用掩码引导的两阶段潜在空间优化。该设计支持子任务并行处理,显著提升运行效率,同时防止失真积累。大量实验表明,Marmot在图像生成任务中显著提升了物体计数、属性赋值及空间关系的准确性。
原文摘要 · Abstract (English)
While diffusion models excel at generating high-quality images, they often struggle with accurate counting, attributes, and spatial relationships in complex multi-object scenes. One potential solution involves employing Multimodal Large Language Model (MLLM) as an AI agent to construct a self-correction framework. However, these approaches heavily rely on the capabilities of the MLLMs used, often fail to account for all objects within the image, and suffer from cumulative distortions during multi-round editing processes. To address these challenges, we propose Marmot, a novel and generalizable framework that leverages Multi-Agent Reasoning for Multi-Object Self-Correcting to enhance image-text alignment. First, we employ a large language model as an Object-Aware Agent to perform object-level divide-and-conquer, automatically decomposing self-correction tasks into object-centric subtasks based on image descriptions. For each subtask, we construct an Object Correction System featuring a decision-execution-verification mechanism that operates exclusively on a single object's segmentation mask or the bounding boxes of object pairs, effectively mitigating inter-object interference and enhancing editing reliability. To efficiently integrate correction results from subtasks while avoiding cumulative distortions from multi-stage editing, we propose a Pixel-Domain Stitching Smoother, which employs mask-guided two-stage latent space optimization. This innovation enables parallel processing of subtasks, significantly improving runtime efficiency while preventing distortion accumulation. Extensive experiments demonstrate that Marmot significantly improves accuracy in object counting, attribute assignment, and spatial relationships for image generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。