用多模态扩散Transformer实现精准创意布局生成。
CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation
- 采用双分支结构分离图文与图布局交互，提升生成可控性。
- 构建含1070万实体的大型布局数据集LayoutSAM，支持高质量训练。
- 集成大模型布局规划器，适合创意设计与UI生成场景。
扩散模型在生成高艺术质量图像方面表现优异。为此，布局到图像（L2I）生成方法被提出，利用区域位置与描述实现更精确可控的生成。然而，现有方法主要基于UNet架构（如SD1.5和SDXL），对多模态扩散Transformer（MM-DiT）的研究较少，尽管后者已展现出强大图像生成能力。将MM-DiT用于L2I看似简单，实则因布局引入、融合与多模态平衡机制复杂而困难。为此，我们探索多种网络结构，最终提出SiamLayout：通过独立权重处理布局，使其与图像、文本同等重要；解耦图像-布局交互为双分支结构，后期再融合，缓解模态间竞争。我们还构建了大规模布局数据集LayoutSAM，包含270万图像-文本对和1070万实体，每个实体附有边界框与详细描述。进一步建立LayoutSAM-Eval评估基准，全面衡量L2I生成质量。最后，推出布局设计师（Layout Designer），利用大语言模型实现智能布局规划与优化。上述组件共同构成CreatiLayout——一个集布局模型、数据集与规划器于一体的系统化创意布局生成方案。
原文摘要 · Abstract (English)
Diffusion models have been recognized for their ability to generate images that are not only visually appealing but also of high artistic quality. As a result, Layout-to-Image (L2I) generation has been proposed to leverage region-specific positions and descriptions to enable more precise and controllable generation. However, previous methods primarily focus on UNet-based models (\eg SD1.5 and SDXL), and limited effort has explored Multimodal Diffusion Transformers (MM-DiTs), which have demonstrated powerful image generation capabilities. Enabling MM-DiT for layout-to-image generation seems straightforward but is challenging due to the complexity of how layout is introduced, integrated, and balanced among multiple modalities. To this end, we explore various network variants to efficiently incorporate layout guidance into MM-DiT, and ultimately present SiamLayout. To inherit the advantages of MM-DiT, we use a separate set of network weights to process the layout, treating it as equally important as the image and text modalities. Meanwhile, to alleviate the competition among modalities, we decouple the image-layout interaction into a siamese branch alongside the image-text one and fuse them in the later stage. Moreover, we contribute a large-scale layout dataset, named LayoutSAM, which includes 2.7 million image-text pairs and 10.7 million entities. Each entity is annotated with a bounding box and a detailed description. We further construct the LayoutSAM-Eval benchmark as a comprehensive tool for evaluating the L2I generation quality. Finally, we introduce the Layout Designer, which taps into the potential of large language models in layout planning, transforming them into experts in layout generation and optimization. These components form CreatiLayout -- a systematic solution that integrates the layout model, dataset, and planner for creative layout-to-image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。