用图结构建模部件关系,让图像生成更符合空间逻辑。
Graph-PiT: Enhancing Structural Coherence in Part-Based Image Synthesis via Graph Priors

- 将视觉部件构建成图,节点表部件,边表空间语义关系。
- 在合成场景中结构一致性提升,真实图片迁移也更合理。
- 适合需要精细控制多部件布局的生成任务,如产品设计。
实现细粒度且结构合理的可控生成是先进视觉生成的核心。现有基于部件的框架将用户提供的部件视为无序集合,忽略其内在的空间与语义关系,常导致组合缺乏结构完整性。为此,我们提出 Graph-PiT,通过图先验显式建模视觉组件的结构性依赖。具体而言,将视觉部件表示为节点,其空间-语义关系表示为边。方法核心为层次化图神经网络(HGNN)模块,在粗粒度部件级超节点与细粒度 IP+ token 子节点间进行双向消息传递,优化部件嵌入后再输入生成流程。同时引入图拉普拉斯平滑损失和边重建损失,使相邻部件获得兼容且关系感知的嵌入。在受控合成域(人物、产品、室内布局、拼图)上的定量实验,以及对真实网页图像的定性迁移结果表明,Graph-PiT 在保持与原始 IP-Prior 流水线兼容的同时,显著提升了结构一致性。消融实验验证了显式关系推理对强制用户指定邻接约束的关键作用。该方法不仅增强了生成概念的合理性,还为复杂多部件图像合成提供了可扩展、可解释的机制。代码已公开于 https://github.com/wolf-bailang/Graph-PiT。
原文摘要 · Abstract (English)
Achieving fine-grained and structurally sound controllability is a cornerstone of advanced visual generation. Existing part-based frameworks treat user-provided parts as an unordered set and therefore ignore their intrinsic spatial and semantic relationships, which often results in compositions that lack structural integrity. To bridge this gap, we propose Graph-PiT, a framework that explicitly models the structural dependencies of visual components using a graph prior. Specifically, we represent visual parts as nodes and their spatial-semantic relationships as edges. At the heart of our method is a Hierarchical Graph Neural Network (HGNN) module that performs bidirectional message passing between coarse-grained part-level super-nodes and fine-grained IP+ token sub-nodes, refining part embeddings before they enter the generative pipeline. We also introduce a graph Laplacian smoothness loss and an edge-reconstruction loss so that adjacent parts acquire compatible, relation-aware embeddings. Quantitative experiments on controlled synthetic domains (character, product, indoor layout, and jigsaw), together with qualitative transfer to real web images, show that Graph-PiT improves structural coherence over vanilla PiT while remaining compatible with the original IP-Prior pipeline. Ablation experiments confirm that explicit relational reasoning is crucial for enforcing user-specified adjacency constraints. Our approach not only enhances the plausibility of generated concepts but also offers a scalable and interpretable mechanism for complex, multi-part image synthesis. The code is available at https://github.com/wolf-bailang/Graph-PiT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。