构建首个关联级多模态删减评估基准，解决敏感信息遗忘难题
SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning
- 基于6万条人格-属性关联数据，设计污染与纯净双模型对比框架
- 提出结构化测试集，精准衡量遗忘效果与副作用，发现现有方法易失效或过度删除
- 开源数据、模型与评测工具，推动可解释删减研究
随着CLIP等多模态模型广泛应用于下游系统，移除敏感信息的需求日益迫切。然而，针对对比学习编码器的机器删减仍缺乏深入研究，现有评估无法诊断细粒度的关联级遗忘。我们提出SALMUBench（敏感关联级多模态删减基准），基于包含6万条人物-属性关联的合成数据集，构建两个基础模型：一个在4亿对保留数据基础上额外训练了敏感数据的污染模型，另一个则未接触该数据的纯净模型。两者均从零开始训练，确保公平对比。我们设计新型评估协议，使用结构化预留集（预留身份、预留关联）精确测量删减效果与附带损伤。实验表明，虽存在高效删减的可能性，但当前方法存在明显失败模式：要么遗忘不彻底，要么过度泛化导致无意删除。SALMUBench为全面删减评估设立新标准，并公开发布数据集、模型、评测脚本与排行榜，以促进未来研究。
原文摘要 · Abstract (English)
As multimodal models like CLIP become integral to downstream systems, the need to remove sensitive information is critical. However, machine unlearning for contrastively-trained encoders remains underexplored, and existing evaluations fail to diagnose fine-grained, association-level forgetting. We introduce SALMUBench (Sensitive Association-Level Multimodal Unlearning), a benchmark built upon a synthetic dataset of 60K persona-attribute associations and two foundational models: a Compromised model polluted with this data, and a Clean model without it. To isolate unlearning effects, both are trained from scratch on the same 400M-pair retain base, with the Compromised model additionally trained on the sensitive set. We propose a novel evaluation protocol with structured holdout sets (holdout identity, holdout association) to precisely measure unlearning efficacy and collateral damage. Our benchmark reveals that while utility-efficient deletion is feasible, current methods exhibit distinct failure modes: they either fail to forget effectively or over-generalize by erasing more than intended. SALMUBench sets a new standard for comprehensive unlearning evaluation, and we publicly release our dataset, models, evaluation scripts, and leaderboards to foster future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。