通过优化输出分布实现大模型精准遗忘,提升隐私保护效果。
Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
- 直接操控模型输出概率分布,替代传统生成响应的遗忘方式。
- 在TOFU基准上遗忘效果最优,在MUSE上保持强泛化能力。
- 无需领域知识,适用于通用场景,适合关注隐私安全的研究者。
大型语言模型(LLMs)虽具备从海量语料中学习的强大能力,但数据隐私与安全问题日益突出。模型遗忘技术旨在移除特定数据的影响,同时保留整体性能,成为重要研究方向。主流方法为基于优化的遗忘方式,如负向偏好优化(NPO),但其受限于缺乏明确的正向偏好信号。现有引入正向信号的方法常需领域知识或精心设计提示,严重限制通用性。本文提出分布偏好优化(DiPO),从分布层面出发,直接针对模型的下一个词概率分布进行优化,而非完整生成响应。通过选择性放大或抑制高置信度输出逻辑值,可构建所需的偏好分布对。理论证明了DiPO损失函数与期望遗忘方向的一致性。大量实验表明,DiPO在遗忘质量与模型性能之间取得良好平衡:在TOFU基准上达到最高遗忘效果,在MUSE基准上保持领先的性能保持能力与可扩展性。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) demonstrate remarkable capabilities learned from vast corpora, concerns regarding data privacy and safety are receiving increasing attention. LLM unlearning, which aims to remove the influence of specific data while preserving overall model utility, is becoming an important research area. One of the mainstream unlearning classes is optimization-based methods, which achieve forgetting directly through fine-tuning, exemplified by Negative Preference Optimization (NPO). However, NPO's effectiveness is limited by its inherent lack of explicit positive preference signals. Attempts to introduce such signals by constructing preferred responses often necessitate domain-specific knowledge or well-designed prompts, fundamentally restricting their generalizability. In this paper, we shift the focus to the distribution-level, directly targeting the next-token probability distribution instead of entire responses, and derive a novel unlearning algorithm termed \textbf{Di}stribution \textbf{P}reference \textbf{O}ptimization (DiPO). We show that the requisite preference distribution pairs for DiPO, which are distributions over the model's output tokens, can be constructed by selectively amplifying or suppressing the model's high-confidence output logits, thereby effectively overcoming NPO's limitations. We theoretically prove the consistency of DiPO's loss function with the desired unlearning direction. Extensive experiments demonstrate that DiPO achieves a strong trade-off between model utility and forget quality. Notably, DiPO attains the highest forget quality on the TOFU benchmark, and maintains leading scalability and sustainability in utility preservation on the MUSE benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。