用图结构生成图像，支持可控、多模态输入。
InstructG2I: Synthesizing Images from Multimodal Attributed Graphs
- 基于图结构与多模态特征采样邻居，增强上下文理解。
- 通过图提示引导扩散模型，实现高质量图像生成。
- 支持调节图引导强度，灵活控制生成结果。
本文提出一个被忽视但重要的任务：从多模态属性图（MMAGs）生成图像。该任务面临图规模爆炸、实体间依赖关系复杂以及生成可控性要求高等挑战。为此，我们提出图上下文条件扩散模型InstructG2I。首先，结合个性化PageRank与视觉-语言特征重排序，高效进行信息丰富邻居采样；其次，采用Graph-QFormer编码器将节点自适应转换为辅助图提示，指导扩散去噪过程；最后，提出图分类器无关引导机制，通过调节图引导强度及节点连接边数，实现可控生成。在三个跨领域数据集上的实验表明，该方法在生成效果与可控性上均表现优异。代码已开源：https://github.com/PeterGriffinJin/InstructG2I。
原文摘要 · Abstract (English)
In this paper, we approach an overlooked yet critical task Graph2Image: generating images from multimodal attributed graphs (MMAGs). This task poses significant challenges due to the explosion in graph size, dependencies among graph entities, and the need for controllability in graph conditions. To address these challenges, we propose a graph context-conditioned diffusion model called InstructG2I. InstructG2I first exploits the graph structure and multimodal information to conduct informative neighbor sampling by combining personalized page rank and re-ranking based on vision-language features. Then, a Graph-QFormer encoder adaptively encodes the graph nodes into an auxiliary set of graph prompts to guide the denoising process of diffusion. Finally, we propose graph classifier-free guidance, enabling controllable generation by varying the strength of graph guidance and multiple connected edges to a node. Extensive experiments conducted on three datasets from different domains demonstrate the effectiveness and controllability of our approach. The code is available at https://github.com/PeterGriffinJin/InstructG2I.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。