针对多模态大模型的持续隐私删除难题,提出新基准ICU-Bench。
ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models

- 构建基于医疗和劳动合同的1000个敏感身份画像数据集
- 在100个连续遗忘任务中发现现有方法难以兼顾历史遗忘与模型能力保留
- 提供考虑历史影响的评估协议,适合隐私敏感场景下的模型迭代测试
隐私删除请求常以顺序方式出现,对已部署的多模态大语言模型(MLLM)构成持续遗忘挑战。现有基准多聚焦于静态或短序列设置,难以评估隐私敏感文档上的持续隐私删除效果。为此,我们提出ICU-Bench——一个以身份为中心的持续遗忘基准(发音为“我看见你”),用于隐私敏感的多模态文档。ICU-Bench包含1,000个来自医学报告和劳动合同的合成隐私敏感身份画像,涵盖9,500张文档图像、16,000个问答对及100个连续遗忘任务。我们进一步引入考虑历史的评估协议和序列感知指标,以衡量当前遗忘、历史遗忘保持、保留效用与稳定性。在两个MLLM上对代表性遗忘方法的实验表明,虽部分方法在当前目标上表现良好,却常无法长期维持早期遗忘效果或保留能力;某些方法甚至通过严重模型退化实现低遗忘准确率。这些结果揭示了传统评估忽略的长周期失效模式,凸显了需专门设计用于持续隐私删除的多模态遗忘方法。代码与基准已开源:https://github.com/AstorYH/ICU-Bench。
原文摘要 · Abstract (English)
Privacy deletion requests often arrive sequentially, creating a continual unlearning challenge for deployed multimodal large language models (MLLMs). However, existing benchmarks mainly focus on static or short-sequence settings, offering limited support for evaluating continual privacy deletion on privacy-critical documents. To bridge this gap, we introduce ICU-Bench, an Identity-centric Continual Unlearning benchmark (pronounced ``I see you'') for privacy-critical multimodal documents. ICU-Bench contains 1,000 synthetic privacy-sensitive profiles from medical reports and labor contracts, comprising 9,500 document images, 16,000 question-answer pairs, and 100 sequential forget tasks. We further introduce history-aware evaluation protocols and sequence-aware metrics to assess current forgetting, historical forgetting preservation, retained utility, and stability throughout the unlearning sequence. Experiments with representative unlearning methods on two MLLMs reveal that methods effective on current targets often fail to preserve earlier forgetting or retained capabilities over long sequences. Some methods further obtain low forget accuracy through severe model degradation. These results expose long-horizon failure modes overlooked by conventional evaluations and highlight the need for multimodal unlearning methods explicitly designed for continual privacy deletion. Our code and benchmark are publicly available at: https://github.com/AstorYH/ICU-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。