通过推理时的负注意力抑制过拟合,提升个性化图像生成效果。
MINDiff: Mask-Integrated Negative Attention for Controlling Overfitting in Text-to-Image Personalization

- 引入负注意力机制,抑制无关区域中的主体影响。
- 在有限图像下显著降低过拟合,提升文本对齐度。
- 无需重训练,可直接用于现有DreamBooth模型。
大规模文本到图像模型在个性化过程中,当仅用少量图像学习特定主体时,常出现过拟合问题。现有方法如DreamBooth通过类别特定先验保留损失缓解此问题,但需增加训练计算成本,并限制推理阶段用户控制。为此,我们提出掩码集成负注意力扩散(MINDiff)。MINDiff引入负注意力概念,在推理时抑制被遮蔽的无关区域中主体的影响,通过修改交叉注意力机制实现。该方法支持语义控制,改善文本对齐,减少主体在无关区域的主导性。此外,推理时用户可调节尺度参数λ,平衡主体保真度与文本对齐。在DreamBooth模型上的定性和定量实验表明,相比类别特定先验保留损失,MINDiff更有效缓解过拟合。本方法完全在推理阶段运行,不改变模型结构,可直接应用于现有DreamBooth模型而无需重新训练。代码已开源。
原文摘要 · Abstract (English)
In the personalization process of large-scale text-to-image models, overfitting often occurs when learning specific subject from a limited number of images. Existing methods, such as DreamBooth, mitigate this issue through a class-specific prior-preservation loss, which requires increased computational cost during training and limits user control during inference time. To address these limitations, we propose Mask-Integrated Negative Attention Diffusion (MINDiff). MINDiff introduces a novel concept, negative attention, which suppresses the subject's influence in masked irrelevant regions. We achieve this by modifying the cross-attention mechanism during inference. This enables semantic control and improves text alignment by reducing subject dominance in irrelevant regions. Additionally, during the inference time, users can adjust a scale parameter lambda to balance subject fidelity and text alignment. Our qualitative and quantitative experiments on DreamBooth models demonstrate that MINDiff mitigates overfitting more effectively than class-specific prior-preservation loss. As our method operates entirely at inference time and does not alter the model architecture, it can be directly applied to existing DreamBooth models without re-training. Our code is available at https://github.com/seuleepy/MINDiff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。