用大模型引导语义对齐,实现通用虚拟试穿。
UniFit: Towards Universal Virtual Try-on with MLLM-Guided Semantic Alignment
- 用多模态大模型+可学习查询对齐文本与图像语义
- 在有限数据下通过两阶段训练实现复杂试穿任务
- 支持多服装、跨模特试穿,效果领先
基于图像的虚拟试穿(VTON)旨在合成人物穿戴指定衣物的逼真图像。尽管进展显著,构建能灵活处理多样复杂任务的通用VTON框架仍是重大挑战。现有方法虽探索了文本指令驱动的多任务框架,但仍存在两大局限:(1)文本指令与参考图像间的语义鸿沟;(2)复杂场景下数据稀缺。为此,我们提出UniFit,一个由多模态大语言模型(MLLM)驱动的通用VTON框架。具体地,引入MLLM引导的语义对齐模块(MGSA),通过MLLM与一组可学习查询融合多模态输入,并施加语义对齐损失,捕捉跨模态语义关系,为生成过程提供连贯明确的语义指导,从而缩小语义鸿沟。此外,设计两阶段渐进式训练策略与自合成流水线,使UniFit能在数据有限情况下学习复杂任务。大量实验表明,UniFit不仅支持多种VTON任务,包括多件衣物和跨模特试穿,且性能达到当前最优。源代码与预训练模型已开源于https://github.com/zwplus/UniFit。
原文摘要 · Abstract (English)
Image-based virtual try-on (VTON) aims to synthesize photorealistic images of a person wearing specified garments. Despite significant progress, building a universal VTON framework that can flexibly handle diverse and complex tasks remains a major challenge. Recent methods explore multi-task VTON frameworks guided by textual instructions, yet they still face two key limitations: (1) semantic gap between text instructions and reference images, and (2) data scarcity in complex scenarios. To address these challenges, we propose UniFit, a universal VTON framework driven by a Multimodal Large Language Model (MLLM). Specifically, we introduce an MLLM-Guided Semantic Alignment Module (MGSA), which integrates multimodal inputs using an MLLM and a set of learnable queries. By imposing a semantic alignment loss, MGSA captures cross-modal semantic relationships and provides coherent and explicit semantic guidance for the generative process, thereby reducing the semantic gap. Moreover, by devising a two-stage progressive training strategy with a self-synthesis pipeline, UniFit is able to learn complex tasks from limited data. Extensive experiments show that UniFit not only supports a wide range of VTON tasks, including multi-garment and model-to-model try-on, but also achieves state-of-the-art performance. The source code and pretrained models are available at https://github.com/zwplus/UniFit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。