无需训练即可精准控制图像中物体数量，通过注意力图聚类实现。
CountCluster: Training-Free Object Quantity Guidance with Cross-Attention Map Clustering for Text-to-Image Generation
- 推理时对物体注意力图进行聚类，按提示词数量划分区域。
- 在早期去噪步骤中使注意力区域空间分离，提升数量准确性18.5%。
- 无需外部计数模块或额外训练，适用于多种提示词场景。
基于扩散模型的文本到图像生成在图像质量和多样性上表现优异，但仍难以准确反映输入提示中指定的物体数量。现有方法依赖外部计数模块或学习到的标记/潜在特征表示，存在数量表达不准的问题，且忽视了关键结构特性：生成图像中的物体实例数量主要由去噪过程的早期阶段决定。为准确反映物体数量，输入提示中指定的数量应与早期去噪阶段物体交叉注意力图中高度激活区域的数量一致，且各区域需清晰分离。为此，我们提出 CountCluster，一种无需外部工具或额外训练的推理时方法，通过注意力得分将物体交叉注意力图划分为 k 个聚类，定义各聚类空间分离的理想分布，并优化潜在表示以匹配该目标分布。实验表明，该方法在物体数量准确率上相比现有方法平均提升18.5%，在多种提示下均展现出优越的数量控制能力。代码将于 https://github.com/JoohyeonL22/CountCluster 公开。
原文摘要 · Abstract (English)
Diffusion-based text-to-image generation models have demonstrated strong performance in terms of image quality and diversity. However, they still struggle to generate images that accurately reflect the number of objects specified in the input prompt. Several approaches have been proposed that rely on either external counting modules for iterative refinement or quantity representations derived from learned tokens or latent features. However, they still have limitations in accurately reflecting the specified number of objects and overlook an important structural characteristic--The number of object instances in the generated image is largely determined in the early timesteps of the denoising process. To correctly reflect the object quantity for image generation, the highly activated regions in the object cross-attention map at the early timesteps should match the input object quantity, while each region should be clearly separated. To address this issue, we propose \textit{CountCluster}, a method that guides the object cross-attention map to be clustered according to the specified object count in the input, without relying on any external tools or additional training. The proposed method partitions the object cross-attention map into $k$ clusters at inference time based on attention scores, defines an ideal distribution in which each cluster is spatially well-separated, and optimizes the latent to align with this target distribution. Our method achieves an average improvement of 18.5\%p in object count accuracy compared to existing methods, and demonstrates superior quantity control performance across a variety of prompts. Code will be released at: https://github.com/JoohyeonL22/CountCluster
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。