用超网络实现无需微调的图像个性化,速度快且效果好。
Finetuning-Free Personalization of Text to Image Generation via Hypernetworks
- 通过超网络直接生成LoRA权重,避免微调
- 在三个数据集上达到强个性化效果,推理速度提升
- 适合需要快速个性化生成的场景
个性化文本到图像扩散模型传统上依赖于如DreamBooth等特定主体微调方法,计算开销大且推理慢。近期基于适配器和编码器的方法虽降低开销,但仍需额外微调或大型主干模型才能获得良好效果。本文重新审视无微调个性化路径:利用超网络直接从主体图像预测LoRA适配权重。针对以往超网络方法存在的数据生成成本高或优化轨迹模拟不稳定的缺陷,我们提出端到端训练目标,并通过简单输出正则化实现稳定,从而获得可靠有效的超网络。该方法在测试时无需对每个主体进行优化,同时保持主体保真度与提示对齐。为进一步提升推理时的组合泛化能力,引入混合模型无分类器引导(HM-CFG),在采样过程中结合基模型的组合优势与个性化模型的主体保真度。在CelebA-HQ、AFHQ-v2和DreamBench上的大量实验表明,本方法实现了强劲的个性化性能,凸显了超网络在开放类别个性化中的可扩展性与有效性。
原文摘要 · Abstract (English)
Personalizing text-to-image diffusion models has traditionally relied on subject-specific fine-tuning approaches such as DreamBooth~\cite{ruiz2023dreambooth}, which are computationally expensive and slow at inference. Recent adapter- and encoder-based methods attempt to reduce this overhead but still depend on additional fine-tuning or large backbone models for satisfactory results. In this work, we revisit an orthogonal direction: fine-tuning-free personalization via Hypernetworks that predict LoRA-adapted weights directly from subject images. Prior hypernetwork-based approaches, however, suffer from costly data generation or unstable attempts to mimic base model optimization trajectories. We address these limitations with an end-to-end training objective, stabilized by a simple output regularization, yielding reliable and effective hypernetworks. Our method removes the need for per-subject optimization at test time while preserving both subject fidelity and prompt alignment. To further enhance compositional generalization at inference time, we introduce Hybrid-Model Classifier-Free Guidance (HM-CFG), which combines the compositional strengths of the base diffusion model with the subject fidelity of personalized models during sampling. Extensive experiments on CelebA-HQ, AFHQ-v2, and DreamBench demonstrate that our approach achieves strong personalization performance and highlights the promise of hypernetworks as a scalable and effective direction for open-category personalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。