让视觉语言模型更安全，同时不损失原有能力。
SafeR-CLIP: Mitigating NSFW Content in Vision-Language Models While Preserving Pre-Trained Knowledge
- 用语义最近的合理目标替换不安全概念，减少模型扰动。
- 在零样本测试中比之前方法准确率提升最高8.0%。
- 适合需要兼顾安全与性能的AI系统开发者使用。
通过微调提升视觉语言模型（如CLIP）的安全性通常会导致泛化性能显著下降。我们发现这一权衡源于僵化的对齐策略：强制不安全概念指向单一预设安全目标，破坏了模型原有的语义结构。为此，我们提出一种邻近感知方法，将不安全概念引导至其语义最接近的安全替代项，以最小化表征变化。我们构建了SaFeR-CLIP微调框架，遵循最小干预原则。该方法成功平衡了安全与性能，在零样本准确率上较先前方法最高恢复8.0%，同时保持强安全性。为支持更严格的评估，我们还提出了NSFW-Caps基准，包含1,000对高度对齐的样本，用于在分布外情况下测试安全性。研究表明，尊重预训练表示的几何结构是实现安全而不牺牲性能的关键。
原文摘要 · Abstract (English)
Improving the safety of vision-language models like CLIP via fine-tuning often comes at a steep price, causing significant drops in their generalization performance. We find this trade-off stems from rigid alignment strategies that force unsafe concepts toward single, predefined safe targets, disrupting the model's learned semantic structure. To address this, we propose a proximity-aware approach: redirecting unsafe concepts to their semantically closest safe alternatives to minimize representational change. We introduce SaFeR-CLIP, a fine-tuning framework that applies this principle of minimal intervention. SaFeR-CLIP successfully reconciles safety and performance, recovering up to 8.0% in zero-shot accuracy over prior methods while maintaining robust safety. To support more rigorous evaluation, we also contribute NSFW-Caps, a new benchmark of 1,000 highly-aligned pairs for testing safety under distributional shift. Our work shows that respecting the geometry of pretrained representations is key to achieving safety without sacrificing performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。