arXiv:2506.00596cs.CV2025-06NeurIPS被引 10

让图像生成精准控形又保语义,支持任意新物体组合

Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control

  • 拆分掩码为语义与轮廓两部分,分别控制内容和形状
  • 在100万张图上训练,实现多物体不混淆的精细生成
  • 适合需要高精度布局设计的AI绘画、广告生成场景

尽管扩散模型取得进展,顶尖文本到图像(T2I)模型仍难以实现精确的空间布局控制,即准确生成具有指定属性和位置的实体。分割掩码到图像(S2I)生成通过引入像素级空间引导和区域文本提示,成为有前景的解决方案。然而,现有S2I方法无法同时保证语义一致性和形状一致性。为此,我们提出Seg2Any,一种基于先进多模态扩散变压器(如FLUX)的新S2I框架。首先,为实现语义与形状一致性,我们将分割掩码条件解耦为区域语义和高频形状成分。区域语义通过语义对齐注意力掩码引入,确保生成实体符合指定文本提示。高频形状成分(即实体边界)以实体轮廓图形式编码,并通过多模态注意力作为额外模态引入,引导图像空间结构。其次,为防止多实体场景中属性泄露,我们引入属性隔离注意力掩码机制,约束每个实体的图像标记仅自注意自身。为支持开放集S2I生成,我们构建了SACap-1M数据集,包含100万张图像、590万段分割实体及详细区域描述,并设立SACap-Eval基准进行综合评估。大量实验表明,Seg2Any在开集与闭集S2I基准上均达到最先进性能,尤其在实体的细粒度空间与属性控制方面表现突出。

原文摘要 · Abstract (English)

Despite recent advances in diffusion models, top-tier text-to-image (T2I) models still struggle to achieve precise spatial layout control, i.e. accurately generating entities with specified attributes and locations. Segmentation-mask-to-image (S2I) generation has emerged as a promising solution by incorporating pixel-level spatial guidance and regional text prompts. However, existing S2I methods fail to simultaneously ensure semantic consistency and shape consistency. To address these challenges, we propose Seg2Any, a novel S2I framework built upon advanced multimodal diffusion transformers (e.g. FLUX). First, to achieve both semantic and shape consistency, we decouple segmentation mask conditions into regional semantic and high-frequency shape components. The regional semantic condition is introduced by a Semantic Alignment Attention Mask, ensuring that generated entities adhere to their assigned text prompts. The high-frequency shape condition, representing entity boundaries, is encoded as an Entity Contour Map and then introduced as an additional modality via multi-modal attention to guide image spatial structure. Second, to prevent attribute leakage across entities in multi-entity scenarios, we introduce an Attribute Isolation Attention Mask mechanism, which constrains each entity's image tokens to attend exclusively to themselves during image self-attention. To support open-set S2I generation, we construct SACap-1M, a large-scale dataset containing 1 million images with 5.9 million segmented entities and detailed regional captions, along with a SACap-Eval benchmark for comprehensive S2I evaluation. Extensive experiments demonstrate that Seg2Any achieves state-of-the-art performance on both open-set and closed-set S2I benchmarks, particularly in fine-grained spatial and attribute control of entities.

图像生成分割控制扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。