arXiv:2510.08132cs.LGcs.AI2025-10NeurIPS被引 3

让视觉语言模型忘记特定图像领域,保留其他领域识别能力。

Approximate Domain Unlearning for Vision-Language Models

  • 通过显式解耦领域分布,动态捕捉实例级领域特征。
  • 在指定领域(如插图)上准确率下降超过30%,其他领域保持稳定。
  • 适用于自动驾驶等需精准区分真实与虚构图像的场景。

预训练的视觉-语言模型(VLMs)具备强大的泛化能力,可无需额外训练即识别多种对象。然而,它们常保留与下游任务无关的信息,引发计算效率和信息泄露问题。这推动了近似遗忘研究的发展,旨在选择性移除冗余知识同时保持整体性能。现有方法主要关注类别遗忘,即让模型无法识别特定物体类别,但实际应用中往往不足。例如,自动驾驶系统应准确识别真实车辆,避免将路边广告中的插画车辆误认为真实车辆。本文提出近似领域遗忘(ADU),要求降低对特定领域(如插图)图像的识别准确率,同时保持对其他领域(如真实图像)的准确率。由于预训练VLM的强领域泛化能力,领域分布高度纠缠于特征空间,基于惩罚目标领域的简单方法无效。为此,我们提出新方法,显式解耦领域分布并自适应捕捉实例级领域信息。大量实验表明,该方法优于基于VLM微调的基线,为视觉-语言模型的实用化、细粒度遗忘开辟了道路。

原文摘要 · Abstract (English)

Pre-trained Vision-Language Models (VLMs) exhibit strong generalization capabilities, enabling them to recognize a wide range of objects across diverse domains without additional training. However, they often retain irrelevant information beyond the requirements of specific downstream tasks, raising concerns about computational efficiency and potential information leakage. This has motivated growing interest in approximate unlearning, which aims to selectively remove unnecessary knowledge while preserving overall model performance. Existing approaches to approximate unlearning have primarily focused on class unlearning, where a VLM is retrained to fail to recognize specified object classes while maintaining accuracy for others. However, merely forgetting object classes is often insufficient in practical applications. For instance, an autonomous driving system should accurately recognize real cars while avoiding misrecognition of illustrated cars depicted in roadside advertisements as real cars, which could be hazardous. In this paper, we introduce Approximate Domain Unlearning (ADU), a novel problem setting that requires reducing recognition accuracy for images from specified domains (e.g., illustration) while preserving accuracy for other domains (e.g., real). ADU presents new technical challenges: due to the strong domain generalization capability of pre-trained VLMs, domain distributions are highly entangled in the feature space, making naive approaches based on penalizing target domains ineffective. To tackle this limitation, we propose a novel approach that explicitly disentangles domain distributions and adaptively captures instance-specific domain information. Extensive experiments show that our approach outperforms baselines built upon VLM tuning techniques, paving the way for practical and fine-grained unlearning in VLMs. Code: https://kodaikawamura.github.io/Domain_Unlearning/.

领域遗忘视觉语言模型知识清理自适应学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。