揭示机器遗忘的内存开销本质,发现现有理论无法准确预测存储需求。
The Space Complexity of Learning-Unlearning Algorithms
- 提出新下界:遗忘能力所需内存至少等于假设类的可消除维数。
- 证明常数VC维模型仍需Ω(n)比特存储,突破传统认知。
- 对比不同记忆模型,揭示核心理论与实际机制的根本差异。
本文研究提供强数据删除保障的机器遗忘算法的内存复杂度。考虑一个学习任务,初始接收训练数据集,学习后收到任意大小用户的数据删除请求,目标是使系统表现如从未接收被删用户数据。问题在于:为实现后续删除,需多少位存储?聚焦可实现性测试任务(即判断剩余样本是否属于给定假设类\\(\mathcal{H}\\)),我们首先给出负结果:VC维不能刻画遗忘的内存复杂度。构造一个恒定VC维和Littlestone维的假设类,但任何针对该任务的遗忘算法均需存储Ω(n)比特,其中n为初始数据集规模。进一步,证明对任意\\$(\mathcal{H}\\),遗忘所需信息量下界为\\(\mathcal{H}\\)的可消除维数,该值恒不小于VC维。我们在Ghazi等(2023)提出的强化票证内存模型下,给出基于星数的上界。因星数不超过可消除维数,本工作凸显了中心模型与票证模型在机器遗忘中的根本差异。
原文摘要 · Abstract (English)
We study the memory complexity of machine unlearning algorithms that provide strong data deletion guarantees to the users. Formally, consider an algorithm for a particular learning task that initially receives a training dataset. Then, after learning, it receives data deletion requests from a subset of users (of arbitrary size), and the goal of unlearning is to perform the task as if the learner never received the data of deleted users. In this paper, we ask how many bits of storage are needed to be able to delete certain training samples at a later time. We focus on the task of realizability testing, where the goal is to check whether the remaining training samples are realizable within a given hypothesis class \(\mathcal{H}\). Toward that end, we first provide a negative result showing that the VC dimension is not a characterization of the space complexity of unlearning. In particular, we provide a hypothesis class with constant VC dimension (and Littlestone dimension), but for which any unlearning algorithm for realizability testing needs to store \(Ω(n)\)-bits, where \(n\) denotes the size of the initial training dataset. In fact, we provide a stronger separation by showing that for any hypothesis class \(\mathcal{H}\), the amount of information that the learner needs to store, so as to perform unlearning later, is lower bounded by the \textit{eluder dimension} of \(\mathcal{H}\), a combinatorial notion always larger than the VC dimension. We complement the lower bound with an upper bound in terms of the star number of the underlying hypothesis class, albeit in a stronger ticketed-memory model proposed by Ghazi et al. (2023). Since the star number for a hypothesis class is never larger than its Eluder dimension, our work highlights a fundamental separation between central and ticketed memory models for machine unlearning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。