根据事实流行度自适应调整遗忘强度，提升大模型删忆效果
The More Popular, The Harder to Forget: Adaptive Popularity for LLM Unlearning
- 按事实流行度动态调节遗忘梯度，流行度越高遗忘越弱
- 在伪装查询下泄露内容减少5倍，对抗重述下减少1.6倍
- 适合需要精准删忆的场景，如隐私保护或敏感信息净化
流行事实在预训练中被更深刻地记忆，且比罕见事实更难删除。现有大模型删忆方法对所有事实施加相同的梯度压力。本文提出AdaPop（自适应流行度）方法，结合局部标记置信度与基于外部代理（如Wikidata sitelinks、LLM-as-Judge）计算的流行度相关指数，并通过双上升控制器每轮自动调节保留惩罚，实现遗忘-保留的动态平衡。在三个模型族和两个基准测试中，当面对改写查询时，AdaPop泄露的遗忘内容比现有方法少约5倍；在对抗性重构下，泄露减少约1.6倍。内部指标显示，采用本方法后，遗忘集隐藏状态与预删忆模型状态的距离显著增大，而保留集表示保持接近。
原文摘要 · Abstract (English)
Popular facts are memorised more deeply during pretraining and resist removal longer than rare ones, yet existing LLM unlearning methods apply uniform gradient pressure regardless of training-data frequency. We propose the AdaPop (Adaptive Popularity) method, which combines local token confidence with a per-fact popularity-dependent exponent derived from an external proxy (e.g., Wikidata sitelinks, LLM-as-Judge), and automates the forget-retain balance via a dual-ascent controller that adjusts the retain penalty each epoch. Across three model families and two benchmarks, AdaPop leaks ~5x less forgotten content than competing methods under paraphrased queries and ~1.6x less under adversarial reformulations. We support our analysis with internal metrics: under our method, forget-set hidden states move further from the pre-unlearning model's states than under other methods, while retain-set representations remain close.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。