通过局部图像块质量评估,提升无微调个性化图像生成的准确性和一致性。
PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generation
- 基于图像块级别质量评分,改进传统DPO方法
- 在单对象和多对象生成任务中达到当前最佳效果
- 适合追求高效、高质量个性化图像生成的研究者
无微调个性化图像生成可在无需测试时微调的情况下合成定制化图像,因其高效性受到广泛关注。现有方法通常仅采用单一训练阶段和简单的图像重建任务,导致测试时生成的图像质量较低且与参考图像不一致。受近期直接偏好优化(DPO)技术启发,本文提出增加一个额外训练阶段以优化预训练个性化生成模型。然而,传统DPO仅判断整体样本优劣,不适用于个性化图像生成,因为生成图像通常仅在局部图像块上与参考图像不一致。为此,本文提出PatchDPO,通过预训练视觉模型结合自监督训练方法估计生成图像中每个图像块的质量,并采用加权训练策略,奖励高质量块,惩罚低质量块。实验表明,PatchDPO显著提升了多个预训练个性化生成模型的性能,在单对象和多对象个性化图像生成任务上均达到当前最优水平。代码已开源:https://github.com/hqhQAQ/PatchDPO。
原文摘要 · Abstract (English)
Finetuning-free personalized image generation can synthesize customized images without test-time finetuning, attracting wide research interest owing to its high efficiency. Current finetuning-free methods simply adopt a single training stage with a simple image reconstruction task, and they typically generate low-quality images inconsistent with the reference images during test-time. To mitigate this problem, inspired by the recent DPO (i.e., direct preference optimization) technique, this work proposes an additional training stage to improve the pre-trained personalized generation models. However, traditional DPO only determines the overall superiority or inferiority of two samples, which is not suitable for personalized image generation because the generated images are commonly inconsistent with the reference images only in some local image patches. To tackle this problem, this work proposes PatchDPO that estimates the quality of image patches within each generated image and accordingly trains the model. To this end, PatchDPO first leverages the pre-trained vision model with a proposed self-supervised training method to estimate the patch quality. Next, PatchDPO adopts a weighted training approach to train the model with the estimated patch quality, which rewards the image patches with high quality while penalizing the image patches with low quality. Experiment results demonstrate that PatchDPO significantly improves the performance of multiple pre-trained personalized generation models, and achieves state-of-the-art performance on both single-object and multi-object personalized image generation. Our code is available at https://github.com/hqhQAQ/PatchDPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。