保护图像隐私不损画质,还能防模型猜你信息
Who Can See Through You? Adversarial Shielding Against VLM-Based Attribute Inference Attacks
- 在保持图像清晰的前提下,联合优化隐私防护与视觉可用性
- 隐私泄露率低于25%,非隐私信息保留率超88%
- 专为真实社交场景设计,适合需兼顾隐私与体验的用户
随着视觉语言模型(VLMs)广泛应用,基于VLM的属性推断攻击已成为严重隐私威胁,使攻击者能从社交媒体分享的图片中推测用户私密信息。现有防护方法常导致图像质量下降或影响视觉功能,难以平衡隐私保护与用户体验。为此,我们提出一种新方法,在视觉一致性约束下联合优化隐私抑制与功能保全。由于缺乏公开评估数据集,方法间比较困难,我们构建了首个公开基准VPI-COCO,包含522张图像及分层隐私问题与对应非隐私版本,支持细粒度联合评估。在多个VLM上的实验表明,该方法可将隐私泄露率(PAR)降至25%以下,非隐私信息保留率(NPAR)维持在88%以上,同时保持高视觉一致性,并对未见及改写后的隐私问题具有良好泛化能力,证明其在真实VLM部署中的强实用性。
原文摘要 · Abstract (English)
As vision-language models (VLMs) become widely adopted, VLM-based attribute inference attacks have emerged as a serious privacy concern, enabling adversaries to infer private attributes from images shared on social media. This escalating threat calls for dedicated protection methods to safeguard user privacy. However, existing methods often degrade the visual quality of images or interfere with vision-based functions on social media, thereby failing to achieve a desirable balance between privacy protection and user experience. To address this challenge, we propose a novel protection method that jointly optimizes privacy suppression and utility preservation under a visual consistency constraint. While our method is conceptually effective, fair comparisons between methods remain challenging due to the lack of publicly available evaluation datasets. To fill this gap, we introduce VPI-COCO, a publicly available benchmark comprising 522 images with hierarchically structured privacy questions and corresponding non-private counterparts, enabling fine-grained and joint evaluation of protection methods in terms of privacy preservation and user experience. Building upon this benchmark, experiments on multiple VLMs demonstrate that our method effectively reduces PAR below 25%, keeps NPAR above 88%, maintains high visual consistency, and generalizes well to unseen and paraphrased privacy questions, demonstrating its strong practical applicability for real-world VLM deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。