首个多维度隐私评估基准，测试大模型泄露敏感信息风险
Multi-PA: A Multi-perspective Benchmark on Privacy Assessment for Large Vision-Language Models
- 构建涵盖31962样本的多视角隐私评测集
- 21个模型普遍存隐私泄露风险，三类隐私漏洞各异
- 适合关注大模型安全与合规的开发者和研究者
大型视觉语言模型（LVLMs）在多任务中表现优异，但存在显著隐私风险，限制其实际应用。现有隐私评估研究覆盖范围有限，评估维度与隐私类别均不完整。为此，我们提出Multi-PA，一个全面的隐私评估基准，用于衡量LVLMs在隐私感知与隐私泄露方面的防护能力。隐私感知指模型识别输入数据隐私敏感性的能力，隐私泄露则评估模型无意间在输出中暴露隐私信息的风险。我们设计了多种子任务，系统评估模型的隐私保护能力。Multi-PA涵盖26类个人隐私、15类商业秘密、18类国家机密，共31,962个样本。基于此，我们评估了21个开源与2个闭源LVLMs。结果表明，当前LVLMs普遍存在较高隐私泄露风险，不同类型的隐私漏洞表现各异。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) exhibit impressive potential across various tasks but also face significant privacy risks, limiting their practical applications. Current researches on privacy assessment for LVLMs is limited in scope, with gaps in both assessment dimensions and privacy categories. To bridge this gap, we propose Multi-PA, a comprehensive benchmark for evaluating the privacy preservation capabilities of LVLMs in terms of privacy awareness and leakage. Privacy awareness measures the model's ability to recognize the privacy sensitivity of input data, while privacy leakage assesses the risk of the model unintentionally disclosing privacy information in its output. We design a range of sub-tasks to thoroughly evaluate the model's privacy protection offered by LVLMs. Multi-PA covers 26 categories of personal privacy, 15 categories of trade secrets, and 18 categories of state secrets, totaling 31,962 samples. Based on Multi-PA, we evaluate the privacy preservation capabilities of 21 open-source and 2 closed-source LVLMs. Our results reveal that current LVLMs generally pose a high risk of facilitating privacy breaches, with vulnerabilities varying across personal privacy, trade secret, and state secret.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。