用反向奖励防御恶意强化学习微调，保护开源模型安全
Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization
- 设计奖励中和机制，让攻击者无法利用有害反馈优化输出
- 仅需50步攻击即可让模型危害度从0-2升至7-9，现有防御无效
- 适合关注开源大模型安全的开发者与研究者
强化学习微调虽能提升大语言模型性能，却暴露了关键漏洞：实验验证，恶意强化学习微调仅需50步及少量对抗提示，即可高效突破安全护栏，使有害性从0-2跃升至7-9。该攻击对具备参数级访问权限的开源模型尤其危险。现有针对监督微调的防御方法在强化学习动态反馈机制面前失效。本文提出首个专为抵御强化学习微调攻击设计的防御框架——奖励中和，通过训练模型生成低信息量拒绝响应，使恶意奖励信号无效化。实验表明，该方法在经历200次攻击后仍保持有害性不超过2，而标准模型迅速恶化。本工作首次证明，对日益普及的强化学习攻击实现鲁棒防御是可行的，填补了开放权重模型的安全空白。
原文摘要 · Abstract (English)
Reinforcement learning (RL) fine-tuning transforms large language models while creating a vulnerability we experimentally verify: Our experiment shows that malicious RL fine-tuning dismantles safety guardrails with remarkable efficiency, requiring only 50 steps and minimal adversarial prompts, with harmful escalating from 0-2 to 7-9. This attack vector particularly threatens open-source models with parameter-level access. Existing defenses targeting supervised fine-tuning prove ineffective against RL's dynamic feedback mechanisms. We introduce Reward Neutralization, the first defense framework specifically designed against RL fine-tuning attacks, establishing concise rejection patterns that render malicious reward signals ineffective. Our approach trains models to produce minimal-information rejections that attackers cannot exploit, systematically neutralizing attempts to optimize toward harmful outputs. Experiments validate that our approach maintains low harmful scores (no greater than 2) after 200 attack steps, while standard models rapidly deteriorate. This work provides the first constructive proof that robust defense against increasingly accessible RL attacks is achievable, addressing a critical security gap for open-weight models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。