让CLIP模型精准删除特定图文关联，不重训也能保持性能。
CLIPErase: Efficient Unlearning of Visual-Textual Associations in CLIP
- 分离视觉与文本关联，针对性清除指定数据
- 在CIFAR-100和Flickr30K上实现零样本遗忘且保留性能
- 适合需合规删数据的多模态应用开发者
机器遗忘（MU）作为一种无需全量重训练即可移除特定数据的方法，近年来受到广泛关注。尽管在文本和图像分类等单模态领域已有进展，多模态模型中的遗忘仍相对未被充分探索。本文针对具有视觉-文本对齐能力的代表性多模态模型CLIP，提出CLIPErase：一种能解耦并选择性遗忘视觉与文本关联的新方法，确保遗忘过程不影响模型整体性能。该方法包含三个模块：遗忘模块破坏目标数据关联，保留模块维持原保留集表现，一致性模块保持与原始模型的一致性。在CIFAR-100与Flickr30K两个数据集上的四项下游任务实验表明，CLIPErase可在零样本场景下有效遗忘指定图文关联，同时在遗忘后保持模型在保留集上的性能。
原文摘要 · Abstract (English)
Machine unlearning (MU) has gained significant attention as a means to remove specific data from trained models without requiring a full retraining process. While progress has been made in unimodal domains like text and image classification, unlearning in multimodal models remains relatively underexplored. In this work, we address the unique challenges of unlearning in CLIP, a prominent multimodal model that aligns visual and textual representations. We introduce CLIPErase, a novel approach that disentangles and selectively forgets both visual and textual associations, ensuring that unlearning does not compromise model performance. CLIPErase consists of three key modules: a Forgetting Module that disrupts the associations in the forget set, a Retention Module that preserves performance on the retain set, and a Consistency Module that maintains consistency with the original model. Extensive experiments on the CIFAR-100 and Flickr30K datasets across four CLIP downstream tasks demonstrate that CLIPErase effectively forgets designated associations in zero-shot tasks for multimodal samples, while preserving the model's performance on the retain set after unlearning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。