arXiv:2602.23391cs.LG2026-02被引 1

通过精准擦除毒性表征,让大模型更彻底地拒绝生成有害内容。

Detoxifying LLMs via Representation Erasure-Based Preference Optimization

  • 将去毒任务转化为逐标记偏好优化,直接修改毒性表征
  • 在多种攻击下仍保持稳定,有效防御重训练和越狱攻击
  • 适合追求模型安全性的研究者与工业应用开发者

基于网络规模数据训练的大语言模型会产生有害输出,引发部署安全担忧。现有基于DPO、NPO等算法的防御方法虽能降低有害续写的概率,但易受对抗性提示攻击,且可被微调重学习攻击轻易破坏。研究表明,这些修改仅停留在表面:线性探测显示,有害“方向”仍存在于模型表征中。为此,本文提出基于表征擦除的偏好优化(REPO),将去毒重构为逐标记层面的偏好问题。利用新颖的目标函数与偏好数据,强制有毒续写表征向其良性对应物收敛。机制分析表明,该细粒度方法至关重要:与基线不同,REPO对毒性编码神经元实施深层、局部修改,同时保留模型整体能力。全面评估显示,REPO实现当前最优鲁棒性,在复杂威胁——包括重训练攻击和增强版GCG越狱——面前表现优异,而现有表征与输出级方法均失效。

原文摘要 · Abstract (English)

Large language models (LLMs) trained on webscale data can produce toxic outputs, raising concerns for safe deployment. Prior defenses, based on applications of DPO, NPO, and similar algorithms, reduce the likelihood of harmful continuations, but not robustly so: they are vulnerable to adversarial prompting and easily undone by fine-tuning-based relearning attacks. Indeed, research has shown that these edits to the model are superficial: linear probing reveals that harmful "directions" remain present in representations. To address this, we propose Representation Erasure-based Preference Optimization (REPO), reformulating detoxification as a token-level preference problem. Using a novel objective with preference data, we force the representations of toxic continuations to converge toward their benign counterparts. Our mechanistic analysis reveals that this granular approach is critical: unlike baselines, REPO induces deep, localized edits to toxicity-encoding neurons while preserving general model utility. Exhaustive evaluations show that REPO achieves state-of-the-art robustness, stopping sophisticated threats-including relearning attacks and enhanced GCG jailbreaks-where existing representation- and output-based methods fail.

大模型安全去毒偏好优化表征擦除

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。