首个评估大模型在复杂隐私场景下保护个人数据能力的基准
IDP-Bench: Benchmarking ability of LLMs to protect personal information in interdependent privacy contexts
- 基于情境完整性框架构建跨层级隐私推理测试
- 八款模型中六款能识别数据共属权，但多数难辨隐私属性与关联主体
- 小模型对隐私判断敏感度高，适合关注隐私安全的研究者参考
大型语言模型作为个人助手广泛接入敏感用户数据，隐私保护成为关键挑战。现有研究多关注个体层面风险，忽视了‘相互依赖隐私’（IDP）——即一人数据可能因他人行为无意识泄露。本文提出首个面向IDP场景的LLM评估基准IDP-Bench，基于情境完整性（CI）框架，在三个推理层级上评估八款开源模型，由两名LLM裁判打分。结果显示：6/8模型对数据共属权识别率超90%，但在识别信息属性、主要主体等CI参数及次级主体等IDP特有参数上表现不佳，7/8模型得分低于74%；判断分享合理性方面，5/8模型低于77%。尽管模型规模增大有助于提升合理性判断能力，但小模型性能下降明显，且提示敏感性仍高，凸显需深入研究大模型在相互依赖隐私中的表现。数据与代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) are becoming widely deployed as personal AI assistants with access to sensitive user data, making privacy a major challenge for their design and evaluation. Prior work focuses mainly on individual-level risks, overlooking \textbf{interdependent privacy (IDP)}--where one person's data may be revealed by others without their knowledge or consent. We address this gap by introducing \textbf{IDP-Bench}: the first LLM benchmark for IDP scenarios, grounded in the Contextual Integrity (CI) framework. We evaluate eight open-source LLMs on their understanding of IDP scenarios across three levels of IDP reasoning using two LLM judges. Results show strong co-ownership recognition (6/8 models exceed 90\%) but persistent weaknesses in identifying CI parameters (information attribute, primary subject) and IDP-specific parameters such as secondary subjects, where 7/8 models score below 74\%. Models also struggle to judge sharing appropriateness (5/8 scoring below 77\%). While the ability to judge the appropriateness of sharing improves with scale, performance tends to decline in smaller models, and prompt sensitivity remains high on IDP-specific questions--highlighting the need for more targeted study of IDP in LLM privacy research. Data \& code available \href{https://github.com/tisl-lab/Interdependent_Privacy_Bench}{here}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。