解决大模型后训练导致的隐私数据检测失效问题。
Calibrating Post-Training Feature Shifts for LLM Data Contamination Detection

- 通过多视角检测后训练引发的特征偏移,定位关键变化。
- 选择性修正特征,提升成员与非成员的区分能力,最高提升15%召回率。
- 适用于各类后训练模型,显著增强数据污染检测可靠性。
大型语言模型(LLM)在大规模且不透明的数据集上训练,可能包含版权或隐私敏感内容。数据污染检测(DCD)旨在判断给定文本是否属于目标模型的预训练语料库。现有先进方法基于特征,从输入文本和模型输出中提取成员特征。然而,现代LLM普遍经历指令微调、偏好优化和推理导向训练等后训练过程,会改变模型输出并引发特征偏移,降低成员与非成员的可分性。为此,我们提出CalibDCD,一种通用的特征基DCD校准框架,包含:(1) 多视角偏移检测,通过在已知非成员文本上测试受控提示变体,识别与后训练相关的重复特征偏移;(2) 有界特征修正,选择性调整与检测到的偏移对齐的特征分量,并控制修正幅度以保留有效检测信息。实验表明,CalibDCD持续提升现有特征基检测器性能,AUC最高提升7.0%,在5%假阳性率下的真阳性率(TPR@5%FPR)提升达15.0%。
原文摘要 · Abstract (English)
Large language models (LLMs) are trained on massive and largely undisclosed corpora that may contain copyrighted or privacy-sensitive content. Data contamination detection (DCD) therefore aims to determine whether a given text is a member of the pre-training corpus of a target LLM. Recent state-of-the-art DCD methods follow a feature-based paradigm that derives membership features from the input text and the corresponding model output. However, most modern LLMs undergo post-training, such as instruction tuning, preference optimization, and reasoning-oriented training, which can alter model outputs and shift the corresponding membership features, thereby reducing the separability between members and non-members. To address this problem, we propose CalibDCD, a broadly applicable calibration framework for feature-based DCD methods, comprising (1) Multi-View Shift Detection, which identifies recurring feature shifts associated with post-training, and (2) Bounded Feature Correction, which selectively mitigates their influence on membership prediction. Specifically, Multi-View Shift Detection evaluates controlled prompt variants on known non-member texts and consolidates the most informative views to identify recurring feature shifts. Bounded Feature Correction selectively adjusts feature components aligned with the detected shifts and controls the correction extent to preserve useful detection information. Experiments show that CalibDCD consistently improves existing feature-based detectors, with gains of up to 7.0% in AUC and 15.0% in TPR@5%FPR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。