让大模型精准删掉特定数据，同时保持整体性能
Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation
- 用动态目标概率实现更精准的自我蒸馏
- 在多个数据集上优于NPO、UnDIAL等前沿方法
- 无需调参，适合实际部署中的隐私合规需求
本文提出Unilogit，一种用于大语言模型机器遗忘的新型自蒸馏方法。针对在遵守GDPR等数据隐私法规下选择性删除特定信息的同时保持模型整体能力的挑战，Unilogit通过动态调整目标词元的概率分布，使其趋于均匀，利用当前模型输出生成更精确的自蒸馏目标。该方法无需额外超参数，显著提升了对理想遗忘目标的逼近能力。在公开基准和内部电商数据集上的大量实验表明，Unilogit在遗忘与保留目标之间实现了更优平衡，性能超越NPO、UnDIAL等现有先进方法。分析进一步验证了其在多种场景下的鲁棒性，凸显了其在实际应用中的有效性与可行性。
原文摘要 · Abstract (English)
This paper introduces Unilogit, a novel self-distillation method for machine unlearning in Large Language Models. Unilogit addresses the challenge of selectively forgetting specific information while maintaining overall model utility, a critical task in compliance with data privacy regulations like GDPR. Unlike prior methods that rely on static hyperparameters or starting model outputs, Unilogit dynamically adjusts target logits to achieve a uniform probability for the target token, leveraging the current model's outputs for more accurate self-distillation targets. This approach not only eliminates the need for additional hyperparameters but also enhances the model's ability to approximate the golden targets. Extensive experiments on public benchmarks and an in-house e-commerce dataset demonstrate Unilogit's superior performance in balancing forget and retain objectives, outperforming state-of-the-art methods such as NPO and UnDIAL. Our analysis further reveals Unilogit's robustness across various scenarios, highlighting its practical applicability and effectiveness in achieving efficacious machine unlearning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。