arXiv:2606.23030cs.CL2026-06

通过提问大模型,可判断其是否学过某实体信息。

Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Models

论文配图:Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Models
图 1 · 摘自论文原文
  • 用提示词诱导模型生成实体相关文本,分析语义特征
  • 在人物实体上实现最高0.97的AUC,准确率提升6%-17.5%
  • 适合关注模型隐私泄露风险的研究者和开发者

大型语言模型(LLMs)引发日益严重的隐私泄露与版权合规担忧。现有研究主要关注特定样本或样本级数据是否被用于训练。我们提出,LLM具有类人记忆行为:虽不完整记住某个样本,但可从零散提及中累积并泄露真实世界实体的知识。基于此,我们提出实体级成员推理,判断目标实体的相关信息是否被用于训练。在仅可观测生成文本的标签仅黑盒设置下,形式化该任务,并建立可行性条件,提出五种基于有限实体线索的提问策略。策略通过构造提示词、获取实体相关回复,利用生成文本间的语义特征推断成员身份。构建实体级数据集,将先进样本级方法适配为基线。实验表明,在人物实体上,本方法达到最高0.97的AUC,平衡准确率较最优基线提升6.0%–17.5%。

原文摘要 · Abstract (English)

Large Language Models (LLMs) raise growing concerns about privacy leakage and copyright compliance. Membership inference is a key tool for assessing such risks, but existing studies mainly focus on whether specific samples or sample-based data units are used for training. We argue that LLMs exhibit a human-memory-like behavior: an LLM may not memorize a specific sample verbatim, yet it can accumulate and reveal knowledge about a real-world entity from scattered mentions. This analogy motivates us to examine whether an LLM can be interrogated like a human interviewee to reveal its exposure to entity-related information. Motivated by this question, we propose entity-level membership inference, which determines whether information related to a target entity is used in LLM training. We study this task in the practical label-only black-box setting, where only generated texts are observable. We formalize the task under clue, input, and model constraints, establish the necessary and sufficient conditions for its feasibility, and instantiate five interrogation strategies based on this formalization. The strategies use limited entity clues to construct prompts, elicit entity-related responses, and infer membership from semantic features among the generated texts. We construct entity-level datasets and adapt state-of-the-art sample-level label-only methods to the entity-level setting as baselines. Experiments on person entities show that our methods achieve AUC up to 0.97 and bring gains of 6.0%--17.5% in Balanced Accuracy over the best adapted baseline.

隐私安全成员推理大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。