用文本摘要对抗恶意文生图提示，提升内容过滤效果
Finding a Wolf in Sheep's Clothing: Combating Adversarial Text-To-Image Prompts with Text Summarization
- 先用模型摘要文本，再分类检测，双重防御恶意提示
- 摘要后分类使F1提升31%，最高达98%
- 适合安全审核、内容过滤系统开发者参考
文生图模型易受分步式‘分而治之攻击’（DACA）威胁，攻击者利用大语言模型将敏感内容包裹在看似无害的叙述中。为应对此类攻击，我们提出两层防护方法：先进行文本摘要，再执行二分类判断。构建了包含940条样本的对抗性文生图提示数据集（ATTIP），涵盖经DACA混淆与未混淆的提示。分别使用小编码器和大语言模型生成摘要版本，并用编码器分类器与GPT-4o对摘要与原始提示进行内容审核。相比直接处理原始文本的分类器，本方法使F1分数提升31%。其中，编码器分类器在摘要版数据上达到最高98%的F1分数。结果表明，预分类摘要可有效抵御分步式混淆攻击。
原文摘要 · Abstract (English)
Text-to-image models are vulnerable to the stepwise "Divide-and-Conquer Attack" (DACA) that utilize a large language model to obfuscate inappropriate content in prompts by wrapping sensitive text in a benign narrative. To mitigate stepwise DACA attacks, we propose a two-layer method involving text summarization followed by binary classification. We assembled the Adversarial Text-to-Image Prompt (ATTIP) dataset ($N=940$), which contained DACA-obfuscated and non-obfuscated prompts. From the ATTIP dataset, we created two summarized versions: one generated by a small encoder model and the other by a large language model. Then, we used an encoder classifier and a GPT-4o classifier to perform content moderation on the summarized and unsummarized prompts. When compared with a classifier that operated over the unsummarized data, our method improved F1 score performance by 31%. Further, the highest recorded F1 score achieved (98%) was produced by the encoder classifier on a summarized ATTIP variant. This study indicates that pre-classification text summarization can inoculate content detection models against stepwise DACA obfuscations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。