arXiv:2505.14832cs.CL2025-05EMNLP被引 6

提出SEPS评估框架,让大模型在混合指令中精准做到该忘的忘、该留的留。

SEPS: A Separability Measure for Robust Unlearning in LLMs

  • 设计混合提示评估框架,模拟真实场景中遗忘与保留问题共存
  • 发现现有方法在多查询时会误删保留内容或彻底失效
  • 提出混合提示训练策略,支持单次提示内最多8个混合查询

机器遗忘旨在有选择性地从大语言模型中移除特定知识,确保其遗忘指定内容的同时保留必要信息。现有遗忘评估指标仅关注模型对保留查询的正确响应和对遗忘查询的拒绝,却无法反映真实场景——遗忘与保留查询常在同一提示中同时出现。为此,我们提出SEPS评估框架,明确衡量模型在单一提示内同时处理遗忘与保留信息的能力。在三个基准上的大量实验揭示了现有遗忘方法的两个关键失败模式:(1) 非目标遗忘会因出现遗忘查询而无差别删除保留内容;(2) 目标遗忘过度适应单查询场景,面对多查询时出现灾难性失效。为解决此问题,我们提出混合提示(MP)遗忘策略,将遗忘与保留查询统一纳入训练目标。该方法显著提升遗忘效果,在包含最多八个混合遗忘与保留查询的复杂提示中仍保持鲁棒性。

原文摘要 · Abstract (English)

Machine unlearning aims to selectively remove targeted knowledge from Large Language Models (LLMs), ensuring they forget specified content while retaining essential information. Existing unlearning metrics assess whether a model correctly answers retain queries and rejects forget queries, but they fail to capture real-world scenarios where forget queries rarely appear in isolation. In fact, forget and retain queries often coexist within the same prompt, making mixed-query evaluation crucial. We introduce SEPS, an evaluation framework that explicitly measures a model's ability to both forget and retain information within a single prompt. Through extensive experiments across three benchmarks, we identify two key failure modes in existing unlearning methods: (1) untargeted unlearning indiscriminately erases both forget and retain content once a forget query appears, and (2) targeted unlearning overfits to single-query scenarios, leading to catastrophic failures when handling multiple queries. To address these issues, we propose Mixed Prompt (MP) unlearning, a strategy that integrates both forget and retain queries into a unified training objective. Our approach significantly improves unlearning effectiveness, demonstrating robustness even in complex settings with up to eight mixed forget and retain queries in a single prompt.

大模型遗忘混合查询评估框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。