用分类器净化提示词，防止生成不当图像。
NSFW-Classifier Guided Prompt Sanitization for Safe Text-to-Image Generation
- 通过文本分类器识别并替换有害词汇，动态净化输入提示。
- 在多个数据集上减少有害内容生成，同时保持图像质量。
- 无需修改模型，适合希望安全使用生成模型的开发者。
文本到图像（T2I）模型（如 Stable Diffusion）的快速发展增强了从文本生成图像的能力，但也带来了滥用风险，例如生成色情、暴力或歧视性内容，违背了T2I技术的伦理目标并阻碍其可持续发展。受大语言模型中‘越狱’攻击启发，本文提出一种无需修改模型架构且不降低生成能力的新型安全机制——NSFW-Classifier Guided Prompt Sanitization（PromptSan）。该方法包含两个变体：PromptSan-Modify 在推理时迭代使用文本NSFW分类器识别并替换提示中的有害词元；PromptSan-Suffix 通过训练一个优化的后缀词序列，在通过文本和图像NSFW分类器检测的同时中和有害意图。大量实验表明，PromptSan 在多个指标上达到当前最优效果，有效平衡安全性与可用性。
原文摘要 · Abstract (English)
The rapid advancement of text-to-image (T2I) models, such as Stable Diffusion, has enhanced their capability to synthesize images from textual prompts. However, this progress also raises significant risks of misuse, including the generation of harmful content (e.g., pornography, violence, discrimination), which contradicts the ethical goals of T2I technology and hinders its sustainable development. Inspired by "jailbreak" attacks in large language models, which bypass restrictions through subtle prompt modifications, this paper proposes NSFW-Classifier Guided Prompt Sanitization (PromptSan), a novel approach to detoxify harmful prompts without altering model architecture or degrading generation capability. PromptSan includes two variants: PromptSan-Modify, which iteratively identifies and replaces harmful tokens in input prompts using text NSFW classifiers during inference, and PromptSan-Suffix, which trains an optimized suffix token sequence to neutralize harmful intent while passing both text and image NSFW classifier checks. Extensive experiments demonstrate that PromptSan achieves state-of-the-art performance in reducing harmful content generation across multiple metrics, effectively balancing safety and usability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。