提出两种方法让语言模型在医疗数据上训练时不记住个人身份信息。
Towards the Anonymization of the Language Modeling
- 用掩码和因果建模避免模型记忆直接与间接身份信息
- 在医疗数据上测试，隐私保护效果优于基线方法
- 适合需共享敏感数据模型的研究者使用
自然语言处理的快速发展改变了多个领域，尤其在医疗中应用广泛。然而，预训练模型在敏感数据上微调后可能记住并泄露个人信息。本文提出一种隐私保护的语言建模方法，促进模型共享。针对BERT类模型，提出掩码语言建模（MLM）方法；针对GPT类模型，提出因果语言建模（CLM）方法，避免模型记忆训练数据中的直接与间接识别信息。我们在医疗数据集上全面评估了该方法，并与多种基线对比。结果表明，在模型专业化过程中避免记忆身份信息，所提出的掩码与因果建模方案能在保障高隐私性的同时维持高模型效用。
原文摘要 · Abstract (English)
Rapid advances in Natural Language Processing (NLP) have revolutionized many fields, including healthcare. However, these advances raise significant privacy concerns, especially when pre-trained models fine-tuned and specialized on sensitive data can memorize and then expose and regurgitate personal information. This paper presents a privacy-preserving language modeling approach to address the problem of language models anonymization, and thus promote their sharing. Specifically, we propose both a Masking Language Modeling (MLM) methodology to specialize a BERT-like language model, and a Causal Language Modeling (CLM) methodology to specialize a GPT-like model that avoids the model from memorizing direct and indirect identifying information present in the training data. We have comprehensively evaluated our approaches using a medical dataset and compared them against different baselines. Our results indicate that by avoiding memorizing both direct and indirect identifiers during model specialization, our masking and causal language modeling schemes offer a good tradeoff for maintaining high privacy while retaining high utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。