用数据学习空间关系损失,让图像生成更懂左右前后。
Data-Driven Loss Functions for Inference-Time Optimization in Text-to-Image
- 从扩散模型注意力图中学习空间关系分类器,作为推理时优化的损失函数。
- 在FLUX.1-dev上空间准确率从20%提升至61%,SD2.1上从7%升至54%。
- 适用于需要精确空间布局的生成任务,如复杂场景合成与视觉推理。
文本到图像扩散模型虽能生成高质量图像,但在处理儿童轻易完成的空间关系任务(如将狗置于泰迪熊右侧而非左侧)时常失败;当组合异常(如长颈鹿位于飞机上方)时,错误更为显著。现有方法依赖人工设计的损失函数进行测试时优化,效果不佳。本文提出Learn-to-Steer框架,不依赖人为假设,而是直接从模型内部表示中学习目标。核心思路是训练一个轻量级分类器,从扩散模型的交叉注意力图中解码空间关系,并在推理时将其作为学习到的损失函数使用。训练中发现,分类器可能通过捕捉提示中的语言线索而非真实空间模式来偷懒。为此,引入包含错误关系词的提示生成样本进行数据增强,迫使分类器关注注意力图中的空间结构。实验显示,该方法显著提升空间准确性:在FLUX.1-dev上从20%提升至61%,在SD2.1上从7%提升至54%;且对多种空间关系具有强泛化能力。
原文摘要 · Abstract (English)
Text-to-image diffusion models can generate stunning visuals, yet they often fail at tasks children find trivial--like placing a dog to the right of a teddy bear rather than to the left. When combinations get more unusual--a giraffe above an airplane--these failures become even more pronounced. Existing methods attempt to fix these spatial reasoning failures through model fine-tuning or test-time optimization with handcrafted losses that are suboptimal. Rather than imposing our assumptions about spatial encoding, we propose learning these objectives directly from the model's internal representations. We introduce Learn-to-Steer, a novel framework that learns data-driven objectives for test-time optimization rather than handcrafting them. Our key insight is to train a lightweight classifier that decodes spatial relationships from the diffusion model's cross-attention maps, then deploy this classifier as a learned loss function during inference. Training such classifiers poses a surprising challenge: they can take shortcuts by detecting linguistic traces in the cross-attention maps, rather than learning true spatial patterns. We solve this by augmenting our training data with samples generated using prompts with incorrect relation words, which encourages the classifier to avoid linguistic shortcuts and learn spatial patterns from the attention maps. Our method dramatically improves spatial accuracy: from 20% to 61% on FLUX.1-dev and from 7% to 54% on SD2.1 across standard benchmarks. It also generalizes to multiple relations with significantly improved accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。