arXiv:2511.07934cs.CV2025-11AAAI被引 7

提升图文生成布局控制精度,保留预训练模型知识。

Laytrol: Preserving Pretrained Knowledge in Layout Control for Multimodal Diffusion Transformers

  • 用继承基模型参数的Laytrol网络实现布局控制
  • 在新数据集上生成图像质量更高且风格一致
  • 适合需要精准布局控制的多模态生成场景

随着扩散模型的发展,提升文本到图像生成中的空间可控性成为关键挑战。布局到图像生成任务旨在生成与给定布局条件空间一致的图像。现有方法通常通过在基础生成模型中引入适配模块来融入布局条件,但生成图像常出现视觉质量低、风格与基模型不一致的问题,表明预训练知识丢失。为此,我们构建了布局合成(LaySyn)数据集,利用基模型自身合成图像以缓解预训练数据分布偏移。同时提出布局控制(Laytrol)网络,其参数源自MM-DiT,以保留基模型的预训练知识。为有效激活复制参数并避免不稳定控制条件干扰,采用专用初始化方案:布局编码器初始化为纯文本编码器,确保输出标记处于MM-DiT数据域内;布局控制网络输出初始化为零。此外,在布局标记上应用对象级旋转位置编码,提供粗粒度位置信息。定性和定量实验验证了方法的有效性。

原文摘要 · Abstract (English)

With the development of diffusion models, enhancing spatial controllability in text-to-image generation has become a vital challenge. As a representative task for addressing this challenge, layout-to-image generation aims to generate images that are spatially consistent with the given layout condition. Existing layout-to-image methods typically introduce the layout condition by integrating adapter modules into the base generative model. However, the generated images often exhibit low visual quality and stylistic inconsistency with the base model, indicating a loss of pretrained knowledge. To alleviate this issue, we construct the Layout Synthesis (LaySyn) dataset, which leverages images synthesized by the base model itself to mitigate the distribution shift from the pretraining data. Moreover, we propose the Layout Control (Laytrol) Network, in which parameters are inherited from MM-DiT to preserve the pretrained knowledge of the base model. To effectively activate the copied parameters and avoid disturbance from unstable control conditions, we adopt a dedicated initialization scheme for Laytrol. In this scheme, the layout encoder is initialized as a pure text encoder to ensure that its output tokens remain within the data domain of MM-DiT. Meanwhile, the outputs of the layout control network are initialized to zero. In addition, we apply Object-level Rotary Position Embedding to the layout tokens to provide coarse positional information. Qualitative and quantitative experiments demonstrate the effectiveness of our method.

布局控制扩散模型多模态生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。