arXiv:2608.04436cs.CV2026-08被引 1

让图像生成模型像人一样用工具思考,自动规划创作全流程。

ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

论文配图:ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation
图 1 · 摘自论文原文
  • 用统一策略动态调度推理、调用外部工具和生成图像
  • 在开放世界任务中比固定流程提升显著,多轮测试准确率超基准32%
  • 适合需要复杂创作的AI设计师、创意工作者使用

文本到图像(T2I)模型虽能生成视觉吸引人的图像,但在需要复杂语义理解、多步推理和外部知识整合的开放世界任务上仍受限。现有方法虽引入代理能力,但或采用固定流程,或仅部分环节由代理控制,导致推理、工具调用与图像生成未协同。我们提出ToolArtist,通过后训练统一多模态模型(UMM)构建全代理式图像生成模型。在监督微调(SFT)阶段,教师代理使用搜索工具和图像生成工具,将收集的轨迹转换为UMM兼容格式,隐藏图像生成工具而保留生成结果。在强化学习(RL)阶段,我们开发了适用于UMM的代理强化学习框架,并提出结合意图与质量奖励的Reason-Act-Draw GRPO(RAD-GRPO)算法,联合优化模型。实验表明,将整个开放世界图像生成过程置于单一代理策略下,持续优于固定流水线或部分代理控制的方法。我们开源了训练数据和完整的后训练基础设施。

原文摘要 · Abstract (English)

Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.

图像生成智能体多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。