只改关键词嵌入,就能精准修正图像生成中的刻板印象。
Implicit Priors Editing in Stable Diffusion via Targeted Token Adjustment
- 仅调整目标词的嵌入向量,不改动模型其他部分。
- 单次编辑仅需修改768或2048个参数,效果提升超6%。
- 适合需要快速、可逆修正生成偏见的实用场景。
文本到图像生成常依赖隐式先验,但这些先验可能包含过时观念或社会偏见。本文提出Embedit方法,仅微调目标词(如'玫瑰')的词嵌入向量(WTE),优化Stable Diffusion文本编码器的最后隐藏状态。该方法不改变无关对象的嵌入和模型权重,确保未涉及目标时输出与原模型一致。单次编辑仅调整768(SD 1.4)或2048(SD XL)个参数,执行迅速且可逆。实验显示,该方法在多种模型和任务中均优于现有方法,性能提升至少6.01%(从87.17%升至93.18%)。
原文摘要 · Abstract (English)
Implicit assumptions and priors are often necessary in text-to-image generation tasks, especially when textual prompts lack sufficient context. However, these assumptions can sometimes reflect outdated concepts, inaccuracies, or societal bias embedded in the training data. We present Embedding-only Editing (Embedit), a method designed to efficiently adjust implict assumptions and priors in the model without affecting its interpretation of unrelated objects or overall performance. Given a "source" prompt (e.g., "rose") that elicits an implicit assumption (e.g., rose is red) and a "destination" prompt that specifies the desired attribute (e.g., "blue rose"), Embedit fine-tunes only the word token embedding (WTE) of the target object ("rose") to optimize the last hidden state of text encoder in Stable Diffusion, a SOTA text-to-image model. This targeted adjustment prevents unintended effects on other objects in the model's knowledge base, as the WTEs for unrelated objects and the model weights remain unchanged. Consequently, when a prompt does not contain the edited object, all representations, and the model outputs are identical to those of the original, unedited model. Our method is highly efficient, modifying only 768 parameters for Stable Diffusion 1.4 and 2048 for XL in a single edit, matching the WTE dimension of each respective model. This minimal scope, combined with rapid execution, makes Embedit highly practical for real-world applications. Additionally, changes are easily reversible by restoring the original WTE layers. Our experimental results demonstrate that Embedit consistently outperforms previous methods across various models, tasks, and editing scenarios (both single and sequential multiple edits), achieving at least a 6.01% improvement (from 87.17% to 93.18%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。