为保护数据隐私,提出可隐藏文本特征的不可学习数据生成方法
Towards Operationalizing Right to Data Protection
- 通过注入难以察觉的虚假关联,使文本数据无法被模型有效学习
- 在小模型和大模型上均验证有效,使GPT-4o、Llama等测试准确率下降
- 无需了解目标模型,适用于公共数据隐私保护场景
大规模无差别爬取数据微调语言模型的做法引发法律与伦理争议,尤其涉及《通用数据保护条例》(GDPR)合规问题。此类行为常导致个人数据未经许可被使用,学术界与监管机构对此日益关注。已有研究提出生成不可学习数据集的方法(如向干净数据添加难以察觉的噪声),使模型训练损失降低但无法泛化至未见测试集。然而,这些方法多针对图像,且受限于需知晓目标模型等实际约束。为此,本文提出RegText框架,通过在自然语言数据中注入难以察觉的虚假相关性,使其在不破坏语义的前提下变得不可学习。我们通过严谨的实证分析验证了RegText在小规模与大规模语言模型上的有效性。值得注意的是,该方法可阻止GPT-4o与Llama等新模型从生成数据中学习,导致其测试准确率低于零样本性能,为保护公众数据提供了生成不可学习文本的新路径。
原文摘要 · Abstract (English)
The widespread practice of indiscriminate data scraping to fine-tune language models (LMs) raises significant legal and ethical concerns, particularly regarding compliance with data protection laws such as the General Data Protection Regulation (GDPR). This practice often results in the unauthorized use of personal information, prompting growing debate within the academic and regulatory communities. Recent works have introduced the concept of generating unlearnable datasets (by adding imperceptible noise to the clean data), such that the underlying model achieves lower loss during training but fails to generalize to the unseen test setting. Though somewhat effective, these approaches are predominantly designed for images and are limited by several practical constraints like requiring knowledge of the target model. To this end, we introduce RegText, a framework that injects imperceptible spurious correlations into natural language datasets, effectively rendering them unlearnable without affecting semantic content. We demonstrate RegText's utility through rigorous empirical analysis of small and large LMs. Notably, RegText can restrict newer models like GPT-4o and Llama from learning on our generated data, resulting in a drop in their test accuracy compared to their zero-shot performance and paving the way for generating unlearnable text to protect public data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。