让同一类物体各不混淆,实现精细的图像生成控制。
InstancePin: Instance-Addressable Layout-to-Image Diffusion via Coordinate Pinning

- 用坐标锚点显式定位每个物体实例,避免类别混淆。
- 在城市街景数据集上,图像质量和语义一致性显著提升。
- 适合需要精细物体分离的应用,如自动驾驶场景生成。
布局到图像的扩散模型通过类别级分割图实现强大的语义控制,但此类控制并非实例可区分:同一类别的多个邻近物体常被当作共享语义区域处理,导致边界模糊、外观平均化及实例间特征混淆。这一限制在密集的都市场景合成中尤为明显,如行人或车辆密集时需细粒度实例分离且保持全局场景一致。本文提出InstancePin,一种实例可区分的布局到图像扩散框架,通过显式坐标锚点为每个物体实例定位。不同于直接注入实例掩码,InstancePin引入独立的实例感知适配器,在保留类别级生成先验的同时学习实例特定的空间控制。每个实例的中心坐标经傅里叶特征编码为坐标标记,由潜在图像特征通过坐标钉扎注意力机制查询。为确保坐标锚点空间有效性,进一步以实例区域监督坐标注意力图,促使每个坐标标记激活对应物体区域。最后,实例掩码引导融合模块将预训练骨干特征路由至非实例区域,适配器特征路由至实例区域,实现局部实例细化而不损失全局语义保真度。在Cityscapes上的大量实验表明,InstancePin有效缓解密集布局中的实例纠缠问题,提升图像保真度与语义一致性。
原文摘要 · Abstract (English)
Layout-to-image diffusion models have achieved impressive semantic controllability by conditioning generation on category-level segmentation maps. However, such category-aligned control is not necessarily instance-addressable: multiple nearby objects from the same category are often treated as a shared semantic region, leading to ambiguous boundaries, averaged appearances, and feature confusion among instances. This limitation is particularly evident in urban scene synthesis, where small and crowded pedestrians or vehicles require fine-grained instance separation while preserving global scene consistency. In this paper, we propose InstancePin, an instance-addressable layout-to-image diffusion framework that pins each object instance with an explicit coordinate anchor. Instead of directly injecting instance masks into the pretrained backbone, InstancePin introduces an independent instance-aware adapter to preserve the category-level generation prior while learning instance-specific spatial control. For each instance, its center coordinate is encoded with Fourier features and projected into a coordinate token, which serves as a spatial anchor queried by latent image features through coordinate pinning attention. To make these anchors spatially meaningful, we further supervise the coordinate attention maps with instance regions, encouraging each coordinate token to activate its corresponding object area. Finally, an instance-mask guided fusion module routes pretrained backbone features to non-instance regions and adapter features to instance regions, enabling local instance refinement without sacrificing global semantic fidelity. Extensive experiments on Cityscapes demonstrate that InstancePin mitigates instance entanglement in dense layouts and improves both image fidelity and semantic consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。