arXiv:2503.14530cs.CVcs.AI2025-03ICCV被引 11

用稀疏自编码器实现视觉语言模型的精准概念删除,不伤及无关知识。

SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders

  • 通过稀疏自编码器捕捉语义丰富的稀疏特征,定位目标概念
  • 在推理时仅修改相关特征,对60个概念的删除效果提升18.04%
  • 支持多概念并行删除,且能抵抗对抗攻击,适用于多种模型

针对视觉语言模型(VLM)的遗忘方法多沿用大语言模型技术,依赖大量标注遗忘数据集,且粒度粗,易造成过度遗忘。为此,我们提出SAUCE,利用稀疏自编码器(SAEs)实现细粒度、选择性概念遗忘。首先训练SAEs以捕获高维语义丰富的稀疏特征;随后识别与目标概念最相关的特征;推理时仅选择性修改这些特征,抑制特定概念而保留无关信息。我们在两个VLM(LLaVA-v1.5-7B和LLAMA-3.2-11B-Vision-Instruct)上评估,涵盖物体、运动场景等具体概念及情绪、颜色、材质等抽象概念,共60个概念。实验表明,SAUCE在遗忘质量上优于现有方法18.04%,同时保持相近模型性能。此外,我们验证了其对常见对抗攻击的鲁棒性、跨模型迁移能力及多请求并发处理的可扩展性,证明SAUCE是高效、可扩展的视觉语言模型选择性遗忘方案。

原文摘要 · Abstract (English)

Unlearning methods for vision-language models (VLMs) have primarily adapted techniques from large language models (LLMs), relying on weight updates that demand extensive annotated forget sets. Moreover, these methods perform unlearning at a coarse granularity, often leading to excessive forgetting and reduced model utility. To address this issue, we introduce SAUCE, a novel method that leverages sparse autoencoders (SAEs) for fine-grained and selective concept unlearning in VLMs. Briefly, SAUCE first trains SAEs to capture high-dimensional, semantically rich sparse features. It then identifies the features most relevant to the target concept for unlearning. During inference, it selectively modifies these features to suppress specific concepts while preserving unrelated information. We evaluate SAUCE on two distinct VLMs, LLaVA-v1.5-7B and LLaMA-3.2-11B-Vision-Instruct, across two types of tasks: concrete concept unlearning (objects and sports scenes) and abstract concept unlearning (emotions, colors, and materials), encompassing a total of 60 concepts. Extensive experiments demonstrate that SAUCE outperforms state-of-the-art methods by 18.04% in unlearning quality while maintaining comparable model utility. Furthermore, we investigate SAUCE's robustness against widely used adversarial attacks, its transferability across models, and its scalability in handling multiple simultaneous unlearning requests. Our findings establish SAUCE as an effective and scalable solution for selective concept unlearning in VLMs.

概念遗忘视觉语言模型稀疏自编码器模型编辑

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。