arXiv:2412.04831cs.CV2024-12ECCV被引 5

提出DCI框架,让图像生成既忠实于概念又精准响应提示。

Customized Generation Reimagined: Fidelity and Editability Harmonized

  • 分治再融合策略,分离并分别优化概念保真与提示响应。
  • 在多个数据集上实现高保真度与强提示对齐的平衡表现。
  • 适合需要精准定制图像生成的设计师与AI创作者使用。

定制化生成旨在将新概念融入预训练文本到图像模型,使模型能在新语境中根据文本提示生成该概念的新图像。然而,定制化生成存在固有的保真度与可编辑性权衡问题,即精确建模概念与忠实遵循提示之间难以兼顾。以往方法被迫妥协,无法同时实现高概念保真度与理想提示对齐。本文提出分治-攻克-整合(DCI)框架,在去噪早期进行手术式调整,使微调模型在推理时摆脱该权衡。通过两个协作分支分别解决两个冲突目标,并选择性融合,从而在保持高概念保真度的同时实现忠实提示响应。为获得更优微调模型,引入图像特定上下文优化(ICO)策略,用可学习的图像特定上下文替代人工提示模板,提供自适应且精确的微调方向,提升整体性能。大量实验表明,该方法有效缓解了保真度与可编辑性的矛盾。

原文摘要 · Abstract (English)

Customized generation aims to incorporate a novel concept into a pre-trained text-to-image model, enabling new generations of the concept in novel contexts guided by textual prompts. However, customized generation suffers from an inherent trade-off between concept fidelity and editability, i.e., between precisely modeling the concept and faithfully adhering to the prompts. Previous methods reluctantly seek a compromise and struggle to achieve both high concept fidelity and ideal prompt alignment simultaneously. In this paper, we propose a Divide, Conquer, then Integrate (DCI) framework, which performs a surgical adjustment in the early stage of denoising to liberate the fine-tuned model from the fidelity-editability trade-off at inference. The two conflicting components in the trade-off are decoupled and individually conquered by two collaborative branches, which are then selectively integrated to preserve high concept fidelity while achieving faithful prompt adherence. To obtain a better fine-tuned model, we introduce an Image-specific Context Optimization} (ICO) strategy for model customization. ICO replaces manual prompt templates with learnable image-specific contexts, providing an adaptive and precise fine-tuning direction to promote the overall performance. Extensive experiments demonstrate the effectiveness of our method in reconciling the fidelity-editability trade-off.

图像生成定制化提示对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。