用强化学习生成难学文本,保护数据不被大模型滥用
TextCloak: Thwarting Unauthorized LLM Exploitation via RL-Driven Unlearnable Text

- 通过强化学习生成保留语义的难学文本
- 在9个大模型上使非法微调效果显著下降
- 适用于多种模型和攻击场景,实用性强
大语言模型的快速发展带来了诸多语言任务的突破,同时也引发了未经授权的数据利用和隐私泄露问题。无学习示例(UEs)通过在数据中引入精心设计的扰动,使模型训练后性能下降,提供了一种有前景的防御方案。然而,现有文本保护方法主要针对分类任务中的判别模型,常依赖特定类别语言线索,难以适用于大模型的开放式生成场景。本文提出TextCloak,一种基于强化学习的文本防护框架。该框架采用生成式策略,将批量原始文本转化为保持语义一致性和语言自然性的难学示例。为优化策略,提出GRPO-UE:根据生成的难学文本在微调的替代大模型中引发的下游性能下降进行奖励,并通过组相对策略优化更新生成器参数。这种双层优化使生成器发现超越类别特异性线索的通用防护模式。在六个公开数据集和九个主流大模型上的全面实验表明,TextCloak能持续削弱未经授权的微调效果,同时保持文本对合法用途的可用性。进一步分析验证了其在不同模型架构、训练配置和自适应攻击下的可迁移性和鲁棒性,凸显其作为实际防御手段的广泛适用性。
原文摘要 · Abstract (English)
The rapid development of Large Language Models (LLMs) has led to significant advances across a wide range of language tasks, while simultaneously raising growing concerns about unauthorized data exploitation and privacy leakage. Unlearnable examples (UEs) offer a promising defense by introducing carefully designed perturbations into data such that models trained on them exhibit degraded utility. However, existing methods for text protection are primarily designed for classification tasks (e.g., sentiment analysis) in discriminative language models and often rely on injecting class-specific linguistic cues, which limits their effectiveness in the open-ended generation settings of LLMs. In this work, we propose TextCloak, an RL-driven framework for protecting textual data against unauthorized LLM exploitation. TextCloak employs a generative policy that transforms batches of clean text into unlearnable examples while preserving semantic fidelity and linguistic naturalness. To optimize the policy, we introduce GRPO-UE, which rewards generated unlearnable text based on the downstream degradation they induce in fine-tuned surrogate LLMs and updates the generator parameters via group-relative policy optimization. This bi-level optimization enables the generator to discover generalizable protective patterns beyond class-specific cues. Comprehensive experiments on six publicly available datasets and nine state-of-the-art LLMs demonstrate that TextCloak consistently impairs unauthorized fine-tuning while maintaining text utility for legitimate use. Further analyses establish its transferability and robustness across model architectures, training configurations, and adaptive attacks, highlighting its broad applicability as a practical defense against unauthorized LLM exploitation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。