用离散概念编码实现大模型零样本高效删忆，可精准擦除敏感信息。
CodeUnlearn: Amortized Zero-Shot Machine Unlearning in Language Models Using Discrete Concept
- 通过码本与稀疏自编码器分解激活空间，实现信息流调控
- 无需重训练即可有效移除特定主题及上下文关联数据
- 首个在大模型中实现带上下文感知删忆的方法，适合隐私保护场景
大型语言模型（LLMs）在多个领域具备广泛知识，但可能无意中记忆敏感、未经授权或恶意数据，如医疗和金融领域的个人信息。机器删忆方法旨在训练后移除模型中的特定信息。然而，现有方法需额外训练，或因LLMs的复杂、密集和连续特性难以有效擦除特定数据点及其上下文。本文提出一种基于码本特征和稀疏自编码器（SAEs）的新型折衷式删忆方法。通过瓶颈结构分解激活空间并调控信息流，该方法能高效移除目标信息，同时保持模型在无关数据上的性能。据我们所知，这是首个成功在LLM中实现带上下文相关性的特定主题删忆的工作，标志着机器删忆向实际应用迈出重要一步。
原文摘要 · Abstract (English)
Large Language Models (LLMs) offer extensive knowledge across various domains, but they may inadvertently memorize sensitive, unauthorized, or malicious data, such as personal information in the medical and financial sectors. Machine unlearning methods aim to remove specific information from models after training to address this. However, current approaches require additional model training or struggle to effectively erase particular data points and their associated context due to LLMs' complex, dense, and continuous nature. In this study, we propose a novel amortized unlearning approach using codebook features and Sparse Autoencoders (SAEs). By leveraging a bottleneck to decompose the activation space and regulate information flow, our method efficiently unlearns targeted information while preserving the model's performance on unrelated data. To the best of our knowledge, this is the first work that successfully enables unlearning specific topics with contextual relevance in an LLM, marking a significant step towards real-world applications of machine unlearning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。