让文字生成图像时精准控制多个物体朝向,支持复杂场景和新物体。
Compass Control: Multi Object Orientation Control for Text-to-Image Generation
- 用方向感知的'指南针'标记符控制每个物体朝向,结合文本输入
- 在合成数据上训练后,可精准控制未见过的复杂物体和三以上物体
- 适合需要精细布局控制的创意设计、3D内容生成场景
现有文本到图像扩散模型虽强大,但无法实现显式的三维物体中心化控制,如精确控制物体朝向。本文提出多物体朝向控制方法,使生成的多物体场景能对每个物体进行精确朝向调控。核心思想是使用一组方向感知的‘指南针’标记符(compass tokens),每个物体一个,并与文本标记符共同作为扩散模型的条件输入。轻量级编码器根据物体朝向预测这些指南针标记符。模型在程序生成的合成场景数据集上训练,每幅图含一个或两个3D资产。然而直接训练导致朝向控制差且物体间产生纠缠。为此,我们干预生成过程,约束每个指南针标记符的交叉注意力映射至对应物体区域。训练后的模型能精确控制:a) 训练中未见的复杂物体;b) 超过两个物体的场景,展现出强泛化能力。结合个性化方法,还能在多样上下文中精确控制新增物体朝向。实验与用户研究显示,本方法在朝向控制和文本对齐方面达到当前最优水平。
原文摘要 · Abstract (English)
Existing approaches for controlling text-to-image diffusion models, while powerful, do not allow for explicit 3D object-centric control, such as precise control of object orientation. In this work, we address the problem of multi-object orientation control in text-to-image diffusion models. This enables the generation of diverse multi-object scenes with precise orientation control for each object. The key idea is to condition the diffusion model with a set of orientation-aware \textbf{compass} tokens, one for each object, along with text tokens. A light-weight encoder network predicts these compass tokens taking object orientation as the input. The model is trained on a synthetic dataset of procedurally generated scenes, each containing one or two 3D assets on a plain background. However, direct training this framework results in poor orientation control as well as leads to entanglement among objects. To mitigate this, we intervene in the generation process and constrain the cross-attention maps of each compass token to its corresponding object regions. The trained model is able to achieve precise orientation control for a) complex objects not seen during training and b) multi-object scenes with more than two objects, indicating strong generalization capabilities. Further, when combined with personalization methods, our method precisely controls the orientation of the new object in diverse contexts. Our method achieves state-of-the-art orientation control and text alignment, quantified with extensive evaluations and a user study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。