无需训练即可精准生成多个实例,解决对象混淆和数量错误问题。
ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation
- 利用自注意力先稳定实例布局,再绑定语义信息
- 在三个评测集上均超越现有无训练方法,准确率提升显著
- 适合需要高精度多实例生成的视觉创作与设计场景
当前开源文本到图像扩散模型在多实例提示下仍存在遗漏或合并实例、语义混杂等问题。我们发现这些失败源于去噪早期阶段,此时实例边界尚未稳定。现有无训练引导主要依赖交叉注意力或令牌条件语义信号,虽可在令牌层面区分概念,但假设实例区域已存在,无法在早期可靠划分区域,导致计数错误和语义混合持续出现。相比之下,自注意力在去噪早期能暴露无类别的实例布局。为此,我们提出ISAC(Instance-to-Semantic Attention Control),一种无需训练、不依赖外部视觉模型的通用目标:先稳定自注意力布局,再在其中绑定交叉注意力语义,无需微调。在T2I-CompBench、HRS-Bench及新构建的IntraCompBench上,ISAC始终优于现有无训练方法。此外,ISAC可提升布局到图像控制器,将粗略重叠的边界框细化为密集实例掩码。代码与IntraCompBench数据集见https://shjo-april.github.io/ISAC。
原文摘要 · Abstract (English)
Recent open-weight text-to-image (T2I) diffusion models still struggle with multi-instance prompts, often omitting or merging instances and mixing semantics among similar objects. We trace these failures to early denoising steps, before instance boundaries are reliably stabilized. Existing training-free guidance is largely driven by cross-attention or other token-conditioned semantic signals. Such guidance can separate concepts at the token level, but largely assumes that distinct instance regions have already emerged. In early denoising steps, it cannot reliably carve out these regions, so count failures and semantic mixing persist. By contrast, self-attention exposes class-agnostic instance layouts during early denoising. To exploit this asymmetry, we propose $\textbf{ISAC}$ ($\textbf{I}$nstance-to-$\textbf{S}$emantic $\textbf{A}$ttention $\textbf{C}$ontrol), a training-free, model-agnostic objective that first stabilizes self-attention layouts and then binds cross-attention semantics within them, without fine-tuning or external vision models. Across T2I-CompBench, HRS-Bench, and our newly curated IntraCompBench, ISAC consistently outperforms prior training-free methods. Furthermore, ISAC enhances layout-to-image controllers by refining coarse, overlapping bounding boxes into dense instance masks. Code and IntraCompBench are available at https://shjo-april.github.io/ISAC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。