提出新损失函数，让大模型更稳定地遗忘敏感数据。
CE-U: Cross Entropy Unlearning
- 用交叉熵统一学习与遗忘，避免梯度消失或爆炸
- 在TOFU基准上优于现有方法，无需额外模型或样本
- 适用于大模型遗忘，对强化学习也有潜在提升
大型语言模型会记忆预训练语料中的敏感数据。本文提出CE-U（交叉熵遗忘）损失函数，解决梯度上升方法在高置信度时梯度消失、低置信度时梯度爆炸的根本问题。同时将标准交叉熵学习与遗忘统一于同一框架。在针对遗忘的TOFU基准测试中，CE-U在不使用额外探针模型或正样本的情况下，实现了LLaMA2-7B模型的当前最优表现。分析表明，类似问题也存在于DPO和GRPO等强化学习算法中，提示将CE-U应用于强化学习可能显著提升稳定性与收敛性。
原文摘要 · Abstract (English)
Large language models memorize sensitive data from their pretraining corpora. In this work, we propose CE-U (Cross Entropy Unlearning), a loss function for unlearning. CE-U addresses fundamental limitations of gradient ascent approaches that suffer from vanishing gradients when model confidence is high and exploding gradients when confidence is low. We also unify standard cross entropy learning and unlearning into a single framework. On the TOFU benchmark for unlearning, CE-U achieves state-of-the-art results on LLaMA2-7B models without using an extra oracle model or additional positive samples. Our analysis reveals that the problematic gradient ascent component also exists in reinforcement learning algorithms like DPO and GRPO. This suggests that applying CE-U approach to reinforcement learning could be promising to improve stability and convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。