通过主动微调让模型重建文本,从而更准识别训练数据来源。
Learning to Detect Language Model Training Data via Active Reconstruction
- 用强化学习主动诱导模型重建候选文本,提升可重构性差异。
- 在多个数据集上检测准确率平均高出10.7%,最高提升18.8%。
- 适合研究模型隐私与数据溯源的学者,尤其关注训练数据泄露问题。
检测大语言模型训练数据通常被视为成员推断攻击(MIA)问题。然而,传统MIA被动依赖固定模型权重,仅使用对数似然或文本生成。本文提出主动数据重建攻击(ADRA),一类主动诱导模型通过微调重建给定文本的MIA方法。我们假设训练数据比非成员数据更具可重构性,其差异可用于成员推断。受强化学习(RL)能强化已有行为的启发,我们采用基于策略的RL,从目标模型初始化策略,主动激发数据重建。为有效应用RL进行MIA,设计了重建指标与对比奖励机制。所得算法 extsc{ADRA}及其自适应变体 extsc{ADRA+}在候选数据池中同时提升重建与检测性能。实验表明,该方法在预训练、后训练及蒸馏数据检测中均优于现有MIA,平均提升10.7%;其中 extsc{ADRA+}在BookMIA上预训练检测优于Min-K extbackslash%++ 18.8%,在AIME上后训练检测提升7.6%。
原文摘要 · Abstract (English)
Detecting LLM training data is generally framed as a membership inference attack (MIA) problem. However, conventional MIAs operate passively on fixed model weights, using log-likelihoods or text generations. In this work, we introduce \textbf{Active Data Reconstruction Attack} (ADRA), a family of MIA that actively induces a model to reconstruct a given text through training. We hypothesize that training data are \textit{more reconstructible} than non-members, and the difference in their reconstructibility can be exploited for membership inference. Motivated by findings that reinforcement learning (RL) sharpens behaviors already encoded in weights, we leverage on-policy RL to actively elicit data reconstruction by finetuning a policy initialized from the target model. To effectively use RL for MIA, we design reconstruction metrics and contrastive rewards. The resulting algorithms, \textsc{ADRA} and its adaptive variant \textsc{ADRA+}, improve both reconstruction and detection given a pool of candidate data. Experiments show that our methods consistently outperform existing MIAs in detecting pre-training, post-training, and distillation data, with an average improvement of 10.7\% over the previous runner-up. In particular, \MethodPlus~improves over Min-K\%++ by 18.8\% on BookMIA for pre-training detection and by 7.6\% on AIME for post-training detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。