不重训练也不用数据,就能精准删除CLIP模型中的特定类别记忆。
Erasing CLIP Memories: Non-Destructive, Data-Free Zero-Shot class Unlearning in CLIP Models
- 通过投影到零空间,直接抹除目标类别的文本嵌入方向。
- 零样本性能下降显著,目标类别识别率大幅降低。
- 适合需要隐私保护或模型去污的场景,效率高且不破坏整体能力。
我们提出一种针对多模态预训练模型(如CLIP)的新型封闭式选择性遗忘方法。该方法利用零空间投影,在不需重新训练或使用遗忘数据图像的情况下,擦除最终投影层中目标类别的信息。通过计算目标文本嵌入张成子空间的正交基并进行投影,显著降低图像特征与非期望类别之间的对齐度。相比依赖迭代微调和大量数据整理的传统方法,本方法计算高效且精准。实验表明,即使部分投影也能在完全遗忘与保留有用信息之间取得平衡,有效解决模型去污与隐私保护的关键挑战。
原文摘要 · Abstract (English)
We introduce a novel, closed-form approach for selective unlearning in multimodal models, specifically targeting pretrained models such as CLIP. Our method leverages nullspace projection to erase the target class information embedded in the final projection layer, without requiring any retraining or the use of images from the forget set. By computing an orthonormal basis for the subspace spanned by target text embeddings and projecting these directions, we dramatically reduce the alignment between image features and undesired classes. Unlike traditional unlearning techniques that rely on iterative fine-tuning and extensive data curation, our approach is both computationally efficient and surgically precise. This leads to a pronounced drop in zero-shot performance for the target classes while preserving the overall multimodal knowledge of the model. Our experiments demonstrate that even a partial projection can balance between complete unlearning and retaining useful information, addressing key challenges in model decontamination and privacy preservation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。