用参考图像指导扩散模型生成，避免特定视觉内容。
Negative Token Merging: Image-based Adversarial Feature Guidance
- 通过图像特征对比，在反向扩散中分离目标与生成特征。
- 使用同批次图像作参考可提升输出多样性，降低版权相似度34.57%。
- 无需训练、代码量少，兼容多种扩散模型，适合快速部署。
基于文本的对抗性引导（如负提示）已成为引导扩散模型避免生成不希望概念的常用方法。然而，仅靠文本难以捕捉复杂视觉概念或规避特定视觉元素（如受版权保护的角色）。本文首次探索以图像为模态的对抗性引导：直接利用参考图像或批内其他图像的视觉特征进行引导。提出一种无需训练的简单有效方法——负标记合并（NegToMe），在反向扩散过程中通过选择性地推离参考图与生成图之间的匹配视觉特征，实现对抗性引导。仅通过更换参考图像，即可支持多样应用。当使用同批次图像作为参考时，显著提升输出多样性（如种族、性别、视觉风格）；当使用受版权保护图像作参考时，使生成结果与原图的视觉相似度降低34.57%。该方法仅需少量代码实现，推理时间增加不足4%，且兼容多种扩散架构（包括不原生支持负提示的Flux模型）。代码已开源。
原文摘要 · Abstract (English)
Text-based adversarial guidance using a negative prompt has emerged as a widely adopted approach to steer diffusion models away from producing undesired concepts. While useful, performing adversarial guidance using text alone can be insufficient to capture complex visual concepts or avoid specific visual elements like copyrighted characters. In this paper, for the first time we explore an alternate modality in this direction by performing adversarial guidance directly using visual features from a reference image or other images in a batch. We introduce negative token merging (NegToMe), a simple but effective training-free approach which performs adversarial guidance through images by selectively pushing apart matching visual features between reference and generated images during the reverse diffusion process. By simply adjusting the used reference, NegToMe enables a diverse range of applications. Notably, when using other images in same batch as reference, we find that NegToMe significantly enhances output diversity (e.g., racial, gender, visual) by guiding features of each image away from others. Similarly, when used w.r.t. copyrighted reference images, NegToMe reduces visual similarity to copyrighted content by 34.57%. NegToMe is simple to implement using just few-lines of code, uses only marginally higher (<4%) inference time and is compatible with different diffusion architectures, including those like Flux, which don't natively support the use of a negative prompt. Code is available at https://negtome.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。