通过知识蒸馏实现私有数据合成，兼顾隐私与模型性能
KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server
- 客户端用差分隐私学习本地知识，服务器蒸馏专业知识
- 在医疗和金融数据上合成文本，性能优于现有方法
- 传输模型而非数据，适合敏感领域应用
大语言模型的兴起使各方可在私有数据上微调模型，但存在记忆风险。现有方案如用合成数据替代，要么依赖本地生成导致性能下降，要么通过API暴露数据。为此，我们提出KnowledgeSG框架，结合差分隐私与知识蒸馏，在客户端学习本地知识、服务器提供专业知识，同时采用联邦学习思想传输模型而非数据。在医疗和金融领域的实验表明该方法能有效提升合成数据质量与模型性能。代码已开源：https://github.com/wwh0411/KnowledgeSG。
原文摘要 · Abstract (English)
The success of large language models (LLMs) facilitate many parties to fine-tune LLMs on their own private data. However, this practice raises privacy concerns due to the memorization of LLMs. Existing solutions, such as utilizing synthetic data for substitution, struggle to simultaneously improve performance and preserve privacy. They either rely on a local model for generation, resulting in a performance decline, or take advantage of APIs, directly exposing the data to API servers. To address this issue, we propose KnowledgeSG, a novel client-server framework which enhances synthetic data quality and improves model performance while ensuring privacy. We achieve this by learning local knowledge from the private data with differential privacy (DP) and distilling professional knowledge from the server. Additionally, inspired by federated learning, we transmit models rather than data between the client and server to prevent privacy leakage. Extensive experiments in medical and financial domains demonstrate the effectiveness of KnowledgeSG. Our code is now publicly available at https://github.com/wwh0411/KnowledgeSG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。