用联邦学习保护隐私,让多个机构协作训练本地RAG模型。
Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation
- 通过联邦学习在不共享数据的前提下协同训练本地检索模型。
- 实验显示新框架显著提升私有RAG性能且保障数据隐私。
- 适合需保护敏感数据的医疗、金融等领域的RAG应用。
检索增强生成(RAG)近期成为提升大语言模型准确性和可信度的有力方案,尤其在问答任务中表现突出,其通过整合专有数据库中的私有数据实现。然而,私有RAG系统面临私域数据稀缺及严峻的数据隐私问题,导致部署困难。开发隐私保护型RAG系统需在数据安全与可用性间取得平衡。为此,我们提出一种基于联邦学习(FL)的新框架——联邦检索增强生成(FedE4RAG),支持客户端RAG检索模型的协同训练。模型参数在中心服务器聚合分发,避免原始数据直接共享。框架中采用知识蒸馏技术促进服务器与客户端模型间的通信,提升本地检索器的泛化能力;同时引入同态加密以保护模型参数,降低数据泄露风险。在真实数据集上的大量实验验证了该框架的有效性,结果表明其可在保持强数据隐私保护的同时,显著提升私有RAG系统的性能。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution for enhancing the accuracy and credibility of Large Language Models (LLMs), particularly in Question & Answer tasks. This is achieved by incorporating proprietary and private data from integrated databases. However, private RAG systems face significant challenges due to the scarcity of private domain data and critical data privacy issues. These obstacles impede the deployment of private RAG systems, as developing privacy-preserving RAG systems requires a delicate balance between data security and data availability. To address these challenges, we regard federated learning (FL) as a highly promising technology for privacy-preserving RAG services. We propose a novel framework called Federated Retrieval-Augmented Generation (FedE4RAG). This framework facilitates collaborative training of client-side RAG retrieval models. The parameters of these models are aggregated and distributed on a central-server, ensuring data privacy without direct sharing of raw data. In FedE4RAG, knowledge distillation is employed for communication between the server and client models. This technique improves the generalization of local RAG retrievers during the federated learning process. Additionally, we apply homomorphic encryption within federated learning to safeguard model parameters and mitigate concerns related to data leakage. Extensive experiments conducted on the real-world dataset have validated the effectiveness of FedE4RAG. The results demonstrate that our proposed framework can markedly enhance the performance of private RAG systems while maintaining robust data privacy protection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。