用对比解码让大模型忘掉特定内容，效果更好且不伤性能。
UCD: Unlearning in LLMs via Contrastive Decoding
- 通过两个辅助小模型的输出差异引导原模型解码。
- 在TOFU和MUSE基准上同时提升遗忘效果与保留性能。
- 适合需要安全可控的LLM应用场景。
机器遗忘旨在从大语言模型中移除特定信息（如敏感或不良内容），同时保持整体性能。我们提出一种推理阶段的遗忘算法，采用对比解码，利用两个辅助小模型——一个未训练包含遗忘数据集，另一个训练包含该数据集——在推理时通过它们输出差异来引导原模型生成。该策略显著改善了遗忘效果与模型可用性之间的权衡。我们在两个遗忘基准测试TOFU和MUSE上评估了该方法，结果表明相比以往方法，在遗忘质量与保留性能方面均有显著提升，说明引入对比解码可为大规模模型遗忘概念提供一种高效、实用的新路径。
原文摘要 · Abstract (English)
Machine unlearning aims to remove specific information, e.g. sensitive or undesirable content, from large language models (LLMs) while preserving overall performance. We propose an inference-time unlearning algorithm that uses contrastive decoding, leveraging two auxiliary smaller models, one trained without the forget set and one trained with it, to guide the outputs of the original model using their difference during inference. Our strategy substantially improves the tradeoff between unlearning effectiveness and model utility. We evaluate our approach on two unlearning benchmarks, TOFU and MUSE. Results show notable gains in both forget quality and retained performance in comparison to prior approaches, suggesting that incorporating contrastive decoding can offer an efficient, practical avenue for unlearning concepts in large-scale models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。