无需训练，用预训练扩散模型实现自由对话式图文生成。
ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers
- 用多智能体系统解析指令并规划生成步骤。
- 在IDEA-Bench上超越有训练的多任务模型。
- 适合创意设计、角色设定等自由交互场景。
近期研究揭示了预训练扩散变换器（DiTs）的上下文生成能力，使其能以极小或无架构修改适配多种视觉任务。通过拼接多输入与目标图像的自注意力令牌，并结合分组与掩码生成流程实现。基于此，我们提出ChatDiT，一个零样本、通用且交互式的视觉生成框架，直接使用原始预训练扩散变换器，无需额外调优、适配器或修改。用户可通过自然语言对话，生成交错文本与图像的文章、多页图画书、编辑图片、设计衍生作品或创建角色设定，支持单轮或多轮交互。核心为三组件多智能体系统：指令解析智能体理解上传图像与指令，策略规划智能体制定单步或多步生成动作，执行智能体利用上下文工具箱中的扩散变换器完成操作。我们在IDEA-Bench（包含100个真实设计任务与275个不同指令及图像数量的案例）上全面评估，尽管方法简单且无训练，其性能仍超越所有对比模型，包括专门在大规模多任务数据集上训练的模型。我们还识别出预训练DiTs在零样本任务适配中的关键局限。代码、智能体、结果与中间输出均已开源，网址：https://github.com/ali-vilab/ChatDiT。
原文摘要 · Abstract (English)
Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the inherent in-context generation capabilities of pretrained diffusion transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks with minimal or no architectural modifications. These capabilities are unlocked by concatenating self-attention tokens across multiple input and target images, combined with grouped and masked generation pipelines. Building upon this foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive visual generation framework that leverages pretrained diffusion transformers in their original form, requiring no additional tuning, adapters, or modifications. Users can interact with ChatDiT to create interleaved text-image articles, multi-page picture books, edit images, design IP derivatives, or develop character design settings, all through free-form natural language across one or more conversational rounds. At its core, ChatDiT employs a multi-agent system comprising three key components: an Instruction-Parsing agent that interprets user-uploaded images and instructions, a Strategy-Planning agent that devises single-step or multi-step generation actions, and an Execution agent that performs these actions using an in-context toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with diverse instructions and varying numbers of input and target images. Despite its simplicity and training-free approach, ChatDiT surpasses all competitors, including those specifically designed and trained on extensive multi-task datasets. We further identify key limitations of pretrained DiTs in zero-shot adapting to tasks. We release all code, agents, results, and intermediate outputs to facilitate further research at https://github.com/ali-vilab/ChatDiT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。