arXiv:2503.19661cs.CV2025-03被引 2

用可控扩散模型一次生成图像和分割掩码,解决标注数据难问题。

CoSimGen: Controllable Diffusion Model for Simultaneous Image and Mask Generation

  • 通过文本、空间和时间嵌入实现图像与掩码的联合控制生成
  • 在4个数据集上达到最低KID 0.11、LPIPS 0.53,性能领先
  • 适合医学影像等需高质量成对数据的领域,支持数据增强

在医学影像、遥感和计算机视觉等领域,获取带有配对图像与分割掩码的标注数据是关键挑战。人工标注耗时耗力,受伦理限制,且依赖专业知识。现有生成模型多仅输出单模态内容(图像或掩码),难以实现高质量的图像-掩码同步生成,且缺乏灵活的条件控制机制,限制了其在数据增强和罕见场景模拟中的应用。本文提出CoSimGen,一种基于扩散模型的可控同步图像与掩码生成框架。通过三类条件实现直观控制:(1) 基于类别语义的文本提示,(2) 提供空间一致性的上下文空间嵌入,(3) 模拟扩散过程中噪声水平的时间步谱嵌入。为提升可控性与训练效率,引入文本与类别嵌入间的对比三元组损失,结合扩散损失与对抗损失。初始生成128×128低分辨率输出,再超分至512×512,生成高保真图像与掩码,并严格遵循输入条件。在FID、KID、LPIPS、Class FID、图像保真度与语义对齐的正预测值等指标上评估,覆盖4个不同数据集。CoSimGen在所有数据集上均达当前最优表现,综合最低KID为0.11,LPIPS为0.53。

原文摘要 · Abstract (English)

The acquisition of annotated datasets with paired images and segmentation masks is a critical challenge in domains such as medical imaging, remote sensing, and computer vision. Manual annotation demands significant resources, faces ethical constraints, and depends heavily on domain expertise. Existing generative models often target single-modality outputs, either images or segmentation masks, failing to address the need for high-quality, simultaneous image-mask generation. Additionally, these models frequently lack adaptable conditioning mechanisms, restricting control over the generated outputs and limiting their applicability for dataset augmentation and rare scenario simulation. We propose CoSimGen, a diffusion-based framework for controllable simultaneous image and mask generation. Conditioning is intuitively achieved through (1) text prompts grounded in class semantics, (2) spatial embedding of context prompts to provide spatial coherence, and (3) spectral embedding of timestep information to model noise levels during diffusion. To enhance controllability and training efficiency, the framework incorporates contrastive triplet loss between text and class embeddings, alongside diffusion and adversarial losses. Initial low-resolution outputs 128 x 128 are super-resolved to 512 x 512, producing high-fidelity images and masks with strict adherence to conditions. We evaluate CoSimGen on metrics such as FID, KID, LPIPS, Class FID, Positive predicted value for image fidelity and semantic alignment of generated samples over 4 diverse datasets. CoSimGen achieves state-of-the-art performance across all datasets, achieving the lowest KID of 0.11 and LPIPS of 0.53 across datasets.

图像生成分割掩码扩散模型可控生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。