arXiv:2412.16978cs.CVcs.AI2024-12ICCV被引 19

用生成式文本提示实现更精准的虚拟试穿,支持风格自由编辑。

PromptDresser: Improving the Quality and Controllability of Virtual Try-On via Generative Textual Prompt and Prompt-aware Mask

  • 通过大模型自动生成服装与人体细节描述,减少人工标注成本。
  • 根据文本动态调整修复掩码,避免旧衣物信息干扰新服装生成。
  • 支持衣着风格灵活修改,适合电商与个性化穿搭场景使用。

近期虚拟试穿方法通过微调预训练文本到图像扩散模型,利用其强大的生成能力取得进展。然而,文本提示在虚拟试穿中的应用仍不充分。本文提出一个可编辑文本的虚拟试穿任务:基于给定服装图像,结合文本描述修改穿着风格(如是否扎进裤腰、松紧度)。该任务面临三大挑战:(i) 为成对的人-衣数据设计丰富文本描述以训练模型;(ii) 解决现有衣物文本信息与新衣物生成之间的冲突;(iii) 自适应调整与文本描述对齐的修复掩码,确保编辑区域准确且保留人物无关部分。为此,我们提出 PromptDresser 模型,借助大模态模型(LMM)通过上下文学习生成高质量、细粒度的文本描述,包括姿态细节与编辑属性,仅需少量人工投入。同时,根据文本提示动态调整修复掩码,提升编辑准确性。实验表明,PromptDresser 显著优于基线模型,在文本驱动控制与多样服装操作上表现优异。代码已开源:https://github.com/rlawjdghek/PromptDresser。

原文摘要 · Abstract (English)

Recent virtual try-on approaches have advanced by finetuning pre-trained text-to-image diffusion models to leverage their powerful generative ability. However, the use of text prompts in virtual try-on remains underexplored. This paper tackles a text-editable virtual try-on task that modifies the clothing based on the provided clothing image while editing the wearing style (e.g., tucking style, fit) according to the text descriptions. In the text-editable virtual try-on, three key aspects exist: (i) designing rich text descriptions for paired person-clothing data to train the model, (ii) addressing the conflicts where textual information of the existing person's clothing interferes the generation of the new clothing, and (iii) adaptively adjust the inpainting mask aligned with the text descriptions, ensuring proper editing areas while preserving the original person's appearance irrelevant to the new clothing. To address these aspects, we propose PromptDresser, a text-editable virtual try-on model that leverages large multimodal model (LMM) assistance to enable high-quality and versatile manipulation based on generative text prompts. Our approach utilizes LMMs via in-context learning to generate detailed text descriptions for person and clothing images independently, including pose details and editing attributes using minimal human cost. Moreover, to ensure the editing areas, we adjust the inpainting mask depending on the text prompts adaptively. Our approach enhances text editability while effectively conveying clothing details that are difficult to capture through images alone, leading to improved image quality. Experiments show that PromptDresser significantly outperforms baselines, demonstrating superior text-driven control and versatile clothing manipulation. Our code is available at https://github.com/rlawjdghek/PromptDresser.

虚拟试穿文本生成扩散模型多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。