arXiv:2409.01086cs.CVcs.AI2024-09被引 2

让服装图像编辑更精准保真,支持文本+图+姿态多模态控制。

DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing

  • 用文本+掩码+姿态+纹理图联合引导生成
  • 引入Grounded-SAM定位编辑区域,精度更高
  • 专设纹理注入机制,保留高细节服装质感

服装图像编辑是设计师可视化创意的重要工具。现有方法虽基于多模态提示和强大扩散模型,但仍难以准确识别编辑区域并保持服装纹理细节。为此,我们提出基于潜在扩散模型的新型多模态服装图像编辑架构——细节保留扩散模型(DPDEdit)。DPDEdit通过整合文本提示、区域掩码、人体姿态图和服装纹理图来引导扩散模型生成。为精确定位编辑区域,我们引入Grounded-SAM,依据用户文本描述预测编辑区域,并与其他条件结合实现局部编辑。为将给定服装纹理细节迁移至目标图像,提出纹理注入与优化机制:采用解耦交叉注意力层融合文本与纹理信息,并引入辅助U-Net以保留生成服装纹理的高频细节。此外,利用多模态大语言模型扩展VITON-HD数据集,生成包含纹理图与文本描述的配对样本。大量实验表明,DPDEdit在图像保真度和与多模态输入的一致性上优于当前最优方法。

原文摘要 · Abstract (English)

Fashion image editing is a crucial tool for designers to convey their creative ideas by visualizing design concepts interactively. Current fashion image editing techniques, though advanced with multimodal prompts and powerful diffusion models, often struggle to accurately identify editing regions and preserve the desired garment texture detail. To address these challenges, we introduce a new multimodal fashion image editing architecture based on latent diffusion models, called Detail-Preserved Diffusion Models (DPDEdit). DPDEdit guides the fashion image generation of diffusion models by integrating text prompts, region masks, human pose images, and garment texture images. To precisely locate the editing region, we first introduce Grounded-SAM to predict the editing region based on the user's textual description, and then combine it with other conditions to perform local editing. To transfer the detail of the given garment texture into the target fashion image, we propose a texture injection and refinement mechanism. Specifically, this mechanism employs a decoupled cross-attention layer to integrate textual descriptions and texture images, and incorporates an auxiliary U-Net to preserve the high-frequency details of generated garment texture. Additionally, we extend the VITON-HD dataset using a multimodal large language model to generate paired samples with texture images and textual descriptions. Extensive experiments show that our DPDEdit outperforms state-of-the-art methods in terms of image fidelity and coherence with the given multimodal inputs.

服装编辑扩散模型多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。