用异构图增强扩散模型图像生成，支持复杂关系条件控制
Heterogeneous Image GNN: Graph-Conditioned Diffusion for Image Synthesis
- 将图像与条件变量建模为双图结构，通过图神经网络融合关系信息
- 在COCO-stuff和Visual Genome上优于现有方法，支持可变长度输入
- 适合需要关系推理的图像生成任务，如场景理解与可控设计
我们提出一种新方法，利用异构图数据对基于扩散的图像生成模型进行条件控制。现有方法通常将条件变量直接嵌入模型架构，如通过跨注意力机制处理文本隐变量或通过图像拼接限制生成空间。然而，这些方法难以处理包含多样、关联性条件变量的复杂场景，而这类信息更自然地表现为无结构图。本文提出异构图像图（HIG），将条件变量与目标图像分别建模为两个互连图，实现对可变长度条件输入及其关系的高效处理。我们还设计了一种保幅值的图神经网络，通过ControlNet方式将HIG集成到现有的EDM2扩散模型中。实验表明，该方法在COCO-stuff和Visual Genome数据集上优于当前最优水平，并展示了对图属性及边所代表关系的有效利用能力。
原文摘要 · Abstract (English)
We introduce a novel method for conditioning diffusion-based image synthesis models with heterogeneous graph data. Existing approaches typically incorporate conditioning variables directly into model architectures, either through cross-attention layers that attend to text latents or image concatenation that spatially restrict generation. However, these methods struggle to handle complex scenarios involving diverse, relational conditioning variables, which are more naturally represented as unstructured graphs. This paper presents Heterogeneous Image Graphs (HIG), a novel representation that models conditioning variables and target images as two interconnected graphs, enabling efficient handling of variable-length conditioning inputs and their relationships. We also propose a magnitude-preserving GNN that integrates the HIG into the existing EDM2 diffusion model using a ControlNet approach. Our approach improves upon the SOTA on a variety of conditioning inputs for the COCO-stuff and Visual Genome datasets, and showcases the ability to condition on graph attributes and relationships represented by edges in the HIG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。