用分步方法提升文生图模型对新主体的保真度，避免遗忘和过拟合。
IMAGE-ALCHEMY: Advancing subject fidelity in personalised text-to-image generation
- 先用原版SDXL生成通用场景，再通过分割驱动的图像修复插入个性化主体
- 在SDXL上实现0.789的DINO相似度，优于现有个性化生成方法
- 适合需要高保真主体生成且资源有限的研究者或创作者
近年来，以Stable Diffusion为代表的文生图扩散模型已能生成高度细节化且语义丰富的图像。然而，基于少量参考图像对模型进行个性化以表征新主体仍具挑战性，常导致灾难性遗忘、过拟合或高计算开销。本文提出一种两阶段流程，通过在Stable Diffusion XL（SDXL）的U-Net注意力权重上采用LoRA微调来解决上述问题。首先，使用未修改的SDXL将主体替换为类别标签生成通用场景；随后，通过分割驱动的图像到图像（Img2Img）管道，利用训练好的LoRA权重选择性插入个性化主体。该框架将主体编码与整体构图分离，从而保留SDXL的整体生成能力，同时以高保真度整合新主体。实验显示，该方法在SDXL上达到0.789的DINO相似度，超越现有个性化文生图方法。
原文摘要 · Abstract (English)
Recent advances in text-to-image diffusion models, particularly Stable Diffusion, have enabled the generation of highly detailed and semantically rich images. However, personalizing these models to represent novel subjects based on a few reference images remains challenging. This often leads to catastrophic forgetting, overfitting, or large computational overhead.We propose a two-stage pipeline that addresses these limitations by leveraging LoRA-based fine-tuning on the attention weights within the U-Net of the Stable Diffusion XL (SDXL) model. First, we use the unmodified SDXL to generate a generic scene by replacing the subject with its class label. Then, we selectively insert the personalized subject through a segmentation-driven image-to-image (Img2Img) pipeline that uses the trained LoRA weights.This framework isolates the subject encoding from the overall composition, thus preserving SDXL's broader generative capabilities while integrating the new subject in a high-fidelity manner. Our method achieves a DINO similarity score of 0.789 on SDXL, outperforming existing personalized text-to-image approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。