提出统一评估指标，解决多模态模型遗忘学习中的评价混乱问题
Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score
- 构建综合评分UQS，融合多种指标权重以提升评估稳定性
- 发现现有五类指标排名冲突，多模态任务中一致性更低
- 适合关注模型合规性与评估标准的AI研究者使用
视觉-语言模型的机器遗忘需满足GDPR要求，但当前评估方法不一致。本文首次系统分析多模态遗忘中的指标可靠性。在三个VQA基准（MLLMU-Bench、UnLOK-VQA、MMUBench）上，五种标准指标（FA、RA、MIA、AD、JS）对36个经过遗忘处理的LLaVA-1.5-7B模型的排名存在冲突。Kendall tau分析显示两组对立集群：{FA, RA, MIA} 和 {AD, JS}，tau_FA_AD = -0.26，该现象在BLIP-2 OPT-2.7B上复现。多模态任务平均tau=0.086，低于单模态分类的0.158（差值=0.072），表明图像与文本双通道加剧不一致。引入统一质量评分UQS，其权重基于各指标与理想模型距离d(M_hat, M_star)的Spearman相关性；其中RA最可靠（rho=0.484, p=0.003），FA则呈负相关（rho=-0.418, p=0.011）。UQS在100次随机权重扰动下保持稳定（tau=0.647±0.262）。论文发布基准数据集、36个检查点及交互式排行榜，代码与预计算结果见https://github.com/neurips26/UnifiedUnl。
原文摘要 · Abstract (English)
Machine unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation (GDPR), yet current evaluation practices are inconsistent. We present the first systematic study of metric reliability in multimodal unlearning. Five standard metrics, Forget Accuracy (FA), Retain Accuracy (RA), Membership Inference Attack (MIA), Activation Distance (AD), and JS divergence (JS), yield conflicting method rankings across three VQA benchmarks (MLLMU-Bench, UnLOK-VQA, MMUBench). Kendall tau analysis over 36 unlearned LLaVA-1.5-7B models reveals two opposing clusters, {FA, RA, MIA} and {AD, JS}, with tau_FA_AD = -0.26, reproduced on BLIP-2 OPT-2.7B. Agreement is lower in multimodal VQA (average tau = 0.086) than in unimodal classification (average tau = 0.158; difference = 0.072), indicating that dual image-and-text pathways amplify inconsistency. We introduce the Unified Quality Score (UQS), a composite metric with weights derived from each metric's Spearman correlation with the oracle distance d(M_hat, M_star), where M_star is the oracle model retrained only on the retain set. RA shows the strongest reliability (rho = 0.484, p = 0.003), while FA is negatively correlated (rho = -0.418, p = 0.011). UQS yields stable rankings under 100 random weight perturbations (tau = 0.647 +- 0.262). We release the benchmark, 36 checkpoints, and an interactive leaderboard. Code and pre-computed results are available at https://github.com/neurips26/UnifiedUnl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。