通过对比解码检测大模型训练数据，提升隐私泄露识别能力
Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding
- 利用成员与非成员上下文的分布差异进行对比解码
- 在WikiMIA基准上达到当前最佳检测性能
- 对文本篡改有强鲁棒性，适合安全审计场景
大型语言模型的训练数据对其成功至关重要，但也可能包含敏感信息，带来隐私和安全风险。检测预训练数据对于缓解此类问题至关重要。现有方法通常孤立分析目标文本或仅依赖非成员上下文，忽略了同时考虑成员与非成员上下文的潜在价值。尽管先前研究认为成员上下文因分布偏移微小而信息量有限，但我们的分析表明，当与非成员上下文对比时，这些细微偏移可被有效利用。本文提出Con-ReCall，通过对比解码放大成员与非成员上下文引起的不对称分布偏移，增强成员推理能力。大量实证评估表明，Con-ReCall在WikiMIA基准上达到当前最优表现，且对多种文本操纵技术具有鲁棒性。
原文摘要 · Abstract (English)
The training data in large language models is key to their success, but it also presents privacy and security risks, as it may contain sensitive information. Detecting pre-training data is crucial for mitigating these concerns. Existing methods typically analyze target text in isolation or solely with non-member contexts, overlooking potential insights from simultaneously considering both member and non-member contexts. While previous work suggested that member contexts provide little information due to the minor distributional shift they induce, our analysis reveals that these subtle shifts can be effectively leveraged when contrasted with non-member contexts. In this paper, we propose Con-ReCall, a novel approach that leverages the asymmetric distributional shifts induced by member and non-member contexts through contrastive decoding, amplifying subtle differences to enhance membership inference. Extensive empirical evaluations demonstrate that Con-ReCall achieves state-of-the-art performance on the WikiMIA benchmark and is robust against various text manipulation techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。