通过融合不同训练阶段的模型，提升个性化图像生成的质量与多样性。
DreamBlend: Advancing Personalized Fine-tuning of Text-to-Image Diffusion Models
- 用早期检查点的提示契合度指导后期检查点的图像合成。
- 在复杂提示下，生成图像的主体还原度、提示契合度和多样性均更优。
- 适合需要高保真个性化图像生成的研究者与应用开发者。
给定少量目标主体的图像，个性化图像生成技术可微调预训练的文本到图像扩散模型，以根据文本提示生成该主体的新场景图像。但在此过程中存在提示契合度、主体契合度与多样性之间的权衡：早期检查点生成的图像提示契合度和多样性高，但主体还原度低；后期检查点生成的图像主体还原度高，但提示契合度和多样性低。本文提出 DreamBlend，推理时结合早期检查点的提示契合度与后期检查点的主体契合度，通过跨注意力机制，以早期检查点生成的图像引导后期检查点的图像合成。该方法在挑战性提示下显著提升生成图像的主体契合度、提示契合度和多样性，优于现有先进微调方法。
原文摘要 · Abstract (English)
Given a small number of images of a subject, personalized image generation techniques can fine-tune large pre-trained text-to-image diffusion models to generate images of the subject in novel contexts, conditioned on text prompts. In doing so, a trade-off is made between prompt fidelity, subject fidelity and diversity. As the pre-trained model is fine-tuned, earlier checkpoints synthesize images with low subject fidelity but high prompt fidelity and diversity. In contrast, later checkpoints generate images with low prompt fidelity and diversity but high subject fidelity. This inherent trade-off limits the prompt fidelity, subject fidelity and diversity of generated images. In this work, we propose DreamBlend to combine the prompt fidelity from earlier checkpoints and the subject fidelity from later checkpoints during inference. We perform a cross attention guided image synthesis from a later checkpoint, guided by an image generated by an earlier checkpoint, for the same prompt. This enables generation of images with better subject fidelity, prompt fidelity and diversity on challenging prompts, outperforming state-of-the-art fine-tuning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。