arXiv:2503.03651cs.CV2025-03CVPR被引 6

用无配对数据让生成模型自适应特定领域,无需标注图文对。

DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cycles

  • 构建双向图文循环,利用统一模型的跨模态映射训练。
  • 仅用无配对数据即可完成风格迁移等任务,少量配对数据可处理身份识别。
  • 适合需要领域定制但标注稀缺的生成任务,如艺术风格迁移。

将生成模型适配到特定领域是满足专业需求的有效方法。然而,在需要大量配对数据才能捕捉目标分布的复杂领域中,适配仍具挑战性。由于单模态(如视觉或语言)的无配对数据更易获取,我们利用统一生成模型学习的视觉-语言双向映射,实现仅基于无配对数据的领域适配。具体地,提出DoraCycle,融合两个多模态循环:文本→图像→文本 和 图像→文本→图像。通过在循环终点计算交叉熵损失(两端同模态),实现模型自进化,无需依赖标注的图文对。实验表明,对于不依赖配对知识的任务(如风格化),仅使用无配对数据即可有效适配统一模型;而对于涉及新配对知识的任务(如特定身份),结合少量配对样本与大规模无配对数据亦可实现有效领域定向适配。代码将开源于 https://github.com/showlab/DoraCycle。

原文摘要 · Abstract (English)

Adapting generative models to specific domains presents an effective solution for satisfying specialized requirements. However, adapting to some complex domains remains challenging, especially when these domains require substantial paired data to capture the targeted distributions. Since unpaired data from a single modality, such as vision or language, is more readily available, we utilize the bidirectional mappings between vision and language learned by the unified generative model to enable training on unpaired data for domain adaptation. Specifically, we propose DoraCycle, which integrates two multimodal cycles: text-to-image-to-text and image-to-text-to-image. The model is optimized through cross-entropy loss computed at the cycle endpoints, where both endpoints share the same modality. This facilitates self-evolution of the model without reliance on annotated text-image pairs. Experimental results demonstrate that for tasks independent of paired knowledge, such as stylization, DoraCycle can effectively adapt the unified model using only unpaired data. For tasks involving new paired knowledge, such as specific identities, a combination of a small set of paired image-text examples and larger-scale unpaired data is sufficient for effective domain-oriented adaptation. The code will be released at https://github.com/showlab/DoraCycle.

生成模型无配对学习领域适配多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。