统一布局规划与图像生成，让模型先想好结构再作图。
PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models
- 用自回归Transformer统一建模布局规划与图像生成
- 支持复杂布局，无需特殊编码即可处理空间条件
- 可扩展至布局引导的图像编辑，适合多任务视觉生成
本文提出统一的布局规划与图像生成模型PlanGen，可在生成图像前预先规划空间布局。不同于以往将布局规划与布局到图像生成分作两个扩散模型的做法，PlanGen将两项任务联合建模于单一自回归Transformer中，仅依赖下一个标记预测。该模型将布局条件作为上下文直接融入，无需对局部描述或边界框坐标进行专门编码，相较于传统的嵌入-池化操作，在处理复杂布局时具有显著优势。统一提示机制使PlanGen能进行多种与布局相关的多任务训练，包括布局规划、布局到图像生成、图像布局理解等。此外，得益于合理建模设计，PlanGen可无缝扩展至布局引导的图像编辑，采用教师强制内容编辑策略与负布局引导机制。大量实验验证了PlanGen在多个布局相关任务中的有效性，展现出巨大潜力。代码已公开：https://360cvgroup.github.io/PlanGen。
原文摘要 · Abstract (English)
In this paper, we propose a unified layout planning and image generation model, PlanGen, which can pre-plan spatial layout conditions before generating images. Unlike previous diffusion-based models that treat layout planning and layout-to-image as two separate models, PlanGen jointly models the two tasks into one autoregressive transformer using only next-token prediction. PlanGen integrates layout conditions into the model as context without requiring specialized encoding of local captions and bounding box coordinates, which provides significant advantages over the previous embed-and-pool operations on layout conditions, particularly when dealing with complex layouts. Unified prompting allows PlanGen to perform multitasking training related to layout, including layout planning, layout-to-image generation, image layout understanding, etc. In addition, PlanGen can be seamlessly expanded to layout-guided image manipulation thanks to the well-designed modeling, with teacher-forcing content manipulation policy and negative layout guidance. Extensive experiments verify the effectiveness of our PlanGen in multiple layoutrelated tasks, showing its great potential. Code is available at: https://360cvgroup.github.io/PlanGen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。