用扩散Transformer生成跨领域的连贯步骤流程,突破数据与一致性难题
MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation
- 基于扩散Transformer和异构LoRA,实现多任务流程生成
- 构建21类任务、超2.4万条流程数据集,提升泛化能力
- 支持从图片反推创作过程,适合工业设计与AI教育应用
人类智能的标志之一是通过结构化的多步骤过程创造复杂成果。用AI生成程序化教程是一项长期但具有挑战性的目标,主要面临三大障碍:(1) 缺乏多任务程序化数据集;(2) 保持步骤间逻辑连贯性与视觉一致性;(3) 在多个领域间的泛化能力。为解决这些问题,我们构建了一个覆盖21个任务、包含超过24,000条程序化序列的多领域数据集。在此基础上,我们提出MakeAnything框架,基于扩散Transformer(DIT),通过微调激活其上下文生成能力,以生成一致的程序化序列。我们引入非对称低秩适应(asymmetric LoRA)用于图像生成,通过冻结编码器参数而自适应调整解码器层,平衡通用性与任务特定性能。此外,我们的ReCraft模型通过时空一致性约束,实现从静态图像生成创作流程,使图像可分解为合理的制作序列。大量实验表明,MakeAnything超越现有方法,在程序化生成任务上设立了新基准。
原文摘要 · Abstract (English)
A hallmark of human intelligence is the ability to create complex artifacts through structured multi-step processes. Generating procedural tutorials with AI is a longstanding but challenging goal, facing three key obstacles: (1) scarcity of multi-task procedural datasets, (2) maintaining logical continuity and visual consistency between steps, and (3) generalizing across multiple domains. To address these challenges, we propose a multi-domain dataset covering 21 tasks with over 24,000 procedural sequences. Building upon this foundation, we introduce MakeAnything, a framework based on the diffusion transformer (DIT), which leverages fine-tuning to activate the in-context capabilities of DIT for generating consistent procedural sequences. We introduce asymmetric low-rank adaptation (LoRA) for image generation, which balances generalization capabilities and task-specific performance by freezing encoder parameters while adaptively tuning decoder layers. Additionally, our ReCraft model enables image-to-process generation through spatiotemporal consistency constraints, allowing static images to be decomposed into plausible creation sequences. Extensive experiments demonstrate that MakeAnything surpasses existing methods, setting new performance benchmarks for procedural generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。