arXiv:2409.08248cs.CV2024-09中稿 · TMLR

只微调文本编码器,高效实现个性化文生图

TextBoost: Boosting Text Encoder for Personalized Text-to-Image Generation

  • 仅微调文本编码器,大幅降低计算与存储开销
  • 保持语义一致性,提升文本匹配度和生成多样性
  • 适合资源受限场景,快速部署高质量个性化生成

本文提出 TextBoost,一种高效的单次个性化方法,用于文生图扩散模型。传统方法需微调大量模型参数,导致存储开销大、收敛慢。TextBoost 仅微调文本编码器,并引入因果性保持的适应机制以保留原始语义。同时,轻量级适配器在文本嵌入与交叉注意力层交互前进行局部优化,显著增强表达力且计算开销极低。实验表明,TextBoost 在多种概念上实现了更快收敛,训练参数量大幅减少,同时保持相近的主体保真度,优于现有方法的文本保真度与生成多样性。该方法为资源受限环境下的高质量个性化生成提供了高效、可扩展的解决方案。

原文摘要 · Abstract (English)

In this paper, we introduce TextBoost, an efficient one-shot personalization approach for text-to-image diffusion models. Traditional personalization methods typically involve fine-tuning extensive portions of the model, leading to substantial storage requirements and slow convergence. In contrast, we propose selectively fine-tuning only the text encoder, significantly improving computational and storage efficiency. To preserve the original semantic integrity, we develop a novel causality-preserving adaptation mechanism. Additionally, lightweight adapters are employed to locally refine text embeddings immediately before their interaction with cross-attention layers, greatly enhancing the expressiveness of text embeddings with minimal computational overhead. Empirical evaluations across diverse concepts demonstrate that TextBoost achieves faster convergence and substantially reduces storage demands by minimizing the number of trainable parameters. Furthermore, TextBoost maintains comparable subject fidelity, superior text fidelity, and greater generation diversity compared to existing methods. We show that our proposed method offers an efficient, scalable, and practically applicable solution for high-quality text-to-image personalization, particularly beneficial in resource-constrained environments.

文生图个性化文本编码器高效微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。