提出自动校准方法，提升大模型成员推理攻击的准确率与可靠性。
Automatic Calibration for Membership Inference Attack on Large Language Models
- 用可调温度校准输出概率，无需额外参考模型。
- 在三个基准上优于现有最先进方法，成员与非成员概率差距更大。
- 适合研究隐私泄露或模型安全的人员使用。
成员推理攻击（MIAs）被用于判断特定文本是否属于大语言模型（LLMs）的预训练数据。然而，现有方法常将非成员误判为成员，导致高假阳性率，或依赖额外参考模型进行概率校准，限制了实用性。为此，我们提出一种新框架——自动校准成员推理攻击（ACMIA），利用可调温度有效校准输出概率。该方法基于对大模型预训练中最大似然估计的理论洞察。我们设计了三种配置，以适应不同模型访问级别，并增强成员与非成员之间的概率差异，从而提升成员推理的可靠性和鲁棒性。在多个开源LLM上的大量实验表明，所提攻击方法高效、稳健且具备良好泛化能力，在三个常用基准上均超越现有最先进基线。代码已开源：https://github.com/Salehzz/ACMIA。
原文摘要 · Abstract (English)
Membership Inference Attacks (MIAs) have recently been employed to determine whether a specific text was part of the pre-training data of Large Language Models (LLMs). However, existing methods often misinfer non-members as members, leading to a high false positive rate, or depend on additional reference models for probability calibration, which limits their practicality. To overcome these challenges, we introduce a novel framework called Automatic Calibration Membership Inference Attack (ACMIA), which utilizes a tunable temperature to calibrate output probabilities effectively. This approach is inspired by our theoretical insights into maximum likelihood estimation during the pre-training of LLMs. We introduce ACMIA in three configurations designed to accommodate different levels of model access and increase the probability gap between members and non-members, improving the reliability and robustness of membership inference. Extensive experiments on various open-source LLMs demonstrate that our proposed attack is highly effective, robust, and generalizable, surpassing state-of-the-art baselines across three widely used benchmarks. Our code is available at: \href{https://github.com/Salehzz/ACMIA}{\textcolor{blue}{Github}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。