arXiv:2605.01735cs.CL2026-05

用少量合成数据实现大模型精准删减,不伤其他能力

Less is More: Geometric Unlearning for LLMs with Minimal Data Disclosure

论文配图:Less is More: Geometric Unlearning for LLMs with Minimal Data Disclosure
图 1 · 摘自论文原文
  • 直接操作隐藏状态,无需原始训练数据
  • 仅用少量安全提示构建安全子空间,精准抑制目标信息
  • 适合需隐私保护的部署场景,对非目标性能影响极小

随着大语言模型在真实系统中广泛应用,必须支持事后移除特定内容以满足隐私与治理要求。这催生了选择性遗忘技术——在保留模型通用能力的前提下,抑制对特定实体或主题的信息。然而,现有方法多依赖原始训练语料,通过输出拒绝调优或广泛梯度更新,导致遗忘强度、非目标保持与数据可用性之间的矛盾。本文提出几何遗忘(Geometric Unlearning, GU),直接作用于提示条件下的隐藏状态,无需访问原始训练语料。具体地,GU 从少量安全参考提示中提炼出紧凑的低秩安全行为子空间,并利用轻量级锚点式合成提示,触发隐藏表示到该安全子空间的局部投影对齐;同时,对合成非目标锚点施加教师蒸馏正则化,进一步减少副作用漂移。在 ToFU 与 UnlearnPII 等面向隐私的遗忘基准上,GU 实现了强目标抑制,且对非目标性能影响极小,证明有效遗忘可在极少合成数据下实现。

原文摘要 · Abstract (English)

As large language models (LLMs) are increasingly deployed in real-world systems, they must support post-hoc removal of specific content to meet privacy and governance requirements. This motivates selective unlearning, which suppresses information about a particular entity or topic while preserving the LLM's general utility. However, most existing LLM unlearning methods require access to the original training corpus and rely on output-level refusal tuning or broad gradient updates, creating a tension among unlearning strength, non-target preservation, and data availability. We propose Geometric Unlearning (GU), an approach that operates directly on the model's prompt-conditioned hidden states without access to the original training corpus. Specifically, GU distills a compact, low-rank safe-behavior subspace from a small set of safe reference prompts and uses lightweight anchor-in-context synthetic prompts to trigger localized, projection-based alignment of hidden representations to this safe subspace. A teacher-distillation regularizer on synthetic non-target anchors further reduces collateral drift. Across privacy-oriented unlearning benchmarks (ToFU and UnlearnPII), GU achieves strong target suppression with minimal impact on non-target performance, demonstrating that effective unlearning can be achieved with minimal synthetic data.

大模型遗忘隐私保护低数据量子空间对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。