首个大规模医学遗忘基准，测试模型如何安全删除患者数据
AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis
- 构建包含8820份病历的7万+问答对医学遗忘基准
- 发现删除单个病人数据会损害同病种其他病人知识
- 适合研究医疗AI隐私保护与知识隔离的学者
医学知识持续演进，亟需更新或选择性遗忘已训练医学大模型中的信息。机器遗忘旨在不重新训练的情况下移除特定训练数据的影响。然而现有遗忘基准依赖合成或小规模通用数据，临床遗忘研究不足。我们提出AMNESIA，首个大规模开源医学遗忘基准，涵盖11种疾病类别、8820份病历生成的70,560个问答对，包含直接回忆与临床推理两类问题。利用该基准评估四种主流遗忘方法在随机患者与疾病层级的表现，并引入新指标检测医学术语泄漏。结果表明，删除个体患者数据会损害相同病症其他患者的模型知识，凸显需开发能更好分离患者与共享临床知识的遗忘方法。
原文摘要 · Abstract (English)
Medical knowledge is continuously evolving. This creates a need to update or selectively forget information encoded in already-trained medical LLMs. Machine unlearning aims to remove the influence of specific training data from a model without full retraining. Yet, existing unlearning benchmarks rely on synthetic or small-scale general data, leaving clinical unlearning understudied. We introduce AMNESIA, the first large-scale, open source benchmark for medical unlearning, with 70,560 question-answer pairs from 8,820 patient notes across 11 disease categories. AMNESIA includes both factual questions testing direct recall and reasoning questions testing clinical inference. We use it to evaluate four widely used unlearning methods at both random patient and disease-level, and introduce a new metric for detecting leakage of medical terminology. We show that unlearning individual patients erodes knowledge of others with the same condition, calling for methods that can better separate patients from shared clinical knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。