新基准揭示大模型成员推断攻击的真实局限性
OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models
- 构建6000张图像的受控数据集，平衡成员与非成员分布
- 顶尖攻击方法成功率接近随机水平，验证隐私风险被高估
- 适合研究模型隐私与安全的学者参考
OpenLVLM-MIA 是一个新提出的基准，揭示了评估大型视觉语言模型（LVLM）成员推断攻击（MIA）时面临的基本挑战。以往研究声称攻击成功率很高，但我们的分析表明，这些结果往往源于数据集构建过程中引入的分布偏倚，而非真实识别成员身份。为此，我们设计了一个包含6,000张图像的受控基准，成员与非成员样本的分布经过精心平衡，并在三个不同训练阶段提供真实成员标签。实验显示，当前最先进的MIA方法性能接近随机水平。OpenLVLM-MIA 作为透明且无偏的基准，澄清了LVLM领域MIA研究的某些局限性，为发展更强的隐私保护技术提供了坚实基础。
原文摘要 · Abstract (English)
OpenLVLM-MIA is a new benchmark that highlights fundamental challenges in evaluating membership inference attacks (MIA) against large vision-language models (LVLMs). While prior work has reported high attack success rates, our analysis suggests that these results often arise from detecting distributional bias introduced during dataset construction rather than from identifying true membership status. To address this issue, we introduce a controlled benchmark of 6{,}000 images where the distributions of member and non-member samples are carefully balanced, and ground-truth membership labels are provided across three distinct training stages. Experiments using OpenLVLM-MIA demonstrated that the performance of state-of-the-art MIA methods approached chance-level. OpenLVLM-MIA, designed to be transparent and unbiased benchmark, clarifies certain limitations of MIA research on LVLMs and provides a solid foundation for developing stronger privacy-preserving techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。