让多实例图像生成更准：位置对，特征也逼真
IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation
- 用外观令牌和语义图控制实例特征位置
- 在新基准上显著提升定位与特征精度
- 可插即用，适配主流扩散模型
虽然文本到图像（T2I）扩散模型能生成单个实例的视觉佳图，但在多个实例的定位与特征控制上表现不佳。布局到图像（L2I）任务通过边界框提供空间控制信号，但仍难以精确生成实例特征。为此，我们提出实例特征生成（IFG）任务，旨在确保生成实例的位置准确性和特征保真度。为此，我们引入实例特征适配器（IFAdapter），通过添加外观令牌并利用实例语义图，将实例级特征与空间位置对齐。IFAdapter作为即插即用模块引导扩散过程，可适配多种社区模型。为评估，我们构建了IFG基准并开发验证流水线，客观比较模型在精准定位和特征生成上的能力。实验结果表明，IFAdapter在定量和定性评估中均优于现有模型。
原文摘要 · Abstract (English)
While Text-to-Image (T2I) diffusion models excel at generating visually appealing images of individual instances, they struggle to accurately position and control the features generation of multiple instances. The Layout-to-Image (L2I) task was introduced to address the positioning challenges by incorporating bounding boxes as spatial control signals, but it still falls short in generating precise instance features. In response, we propose the Instance Feature Generation (IFG) task, which aims to ensure both positional accuracy and feature fidelity in generated instances. To address the IFG task, we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances feature depiction by incorporating additional appearance tokens and utilizing an Instance Semantic Map to align instance-level features with spatial locations. The IFAdapter guides the diffusion process as a plug-and-play module, making it adaptable to various community models. For evaluation, we contribute an IFG benchmark and develop a verification pipeline to objectively compare models' abilities to generate instances with accurate positioning and features. Experimental results demonstrate that IFAdapter outperforms other models in both quantitative and qualitative evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。