用一张图定制扩散模型,避免过拟合,生成更自然。
T-LoRA: Single Image Diffusion Model Customization Without Overfitting
- 按时间步动态调整微调强度,防止过拟合
- 在SD-XL和FLUX-1.dev上实现更高概念保真度
- 适合需要单图个性化生成的开发者
尽管微调扩散模型可有效定制生成特定对象,但在样本有限时常出现过拟合,影响泛化能力和输出多样性。本文针对最具实用价值的单图像定制任务,提出T-LoRA——一种针对扩散模型个性化的时步依赖低秩适配框架。研究发现高时步更易过拟合,需采用时步敏感的微调策略。T-LoRA包含两项创新:(1) 基于扩散时步动态调整秩约束更新;(2) 通过正交初始化确保适配器组件独立性。在SD-XL与FLUX-1.dev上的大量实验表明,T-LoRA及其组件优于标准LoRA及其他个性化方法,在概念保真度与文本对齐间取得更优平衡。
原文摘要 · Abstract (English)
While diffusion model fine-tuning offers a powerful approach for customizing pre-trained models to generate specific objects, it frequently suffers from overfitting when training samples are limited, compromising both generalization capability and output diversity. This paper tackles the challenging yet most impactful task of adapting a diffusion model using just a single concept image, as single-image customization holds the greatest practical potential. We introduce T-LoRA, a Timestep-Dependent Low-Rank Adaptation framework specifically designed for diffusion model personalization. We show that higher diffusion timesteps are more prone to overfitting than lower ones, necessitating a timestep-sensitive fine-tuning strategy. T-LoRA incorporates two key innovations: (1) a dynamic fine-tuning strategy that adjusts rank-constrained updates based on diffusion timesteps, and (2) a weight parametrization technique that ensures independence between adapter components through orthogonal initialization. Extensive experiments on SD-XL and FLUX-1.dev show that T-LoRA and its individual components outperform standard LoRA and other diffusion model personalization techniques, achieving a superior balance between concept fidelity and text alignment. Project page is available at https://controlgenai.github.io/T-LoRA/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。