让模块化大模型在保护隐私的前提下实现跨领域知识迁移。
NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation
- 结合差分隐私与低秩适配器+提示词的两阶段微调方法。
- 在CodeXGLUE上实现可证明的隐私保护，代码补全准确率提升77%。
- 适合关注隐私安全与知识共享的AI研发人员。
大规模语言模型通常在来自多种来源的海量数据上训练，即使采用模块化设计（如专家混合模型），仍可能泄露数据源隐私。而孤立训练又阻碍泛化能力。为此，我们提出NoEsis框架，兼顾模块性、隐私保护与知识迁移。该框架将差分隐私与混合式两阶段参数高效微调结合：使用领域特定的低秩适配器作为专家，共同提示词作为知识共享主干。在CodeXGLUE上的评估表明，NoEsis能实现可证明的隐私保障，并有效抵御成员推理攻击；在代码补全任务中，其性能至少弥补了非共享与非私有基线之间77%的准确率差距。
原文摘要 · Abstract (English)
Large Language Models (LLM) are typically trained on vast amounts of data from various sources. Even when designed modularly (e.g., Mixture-of-Experts), LLMs can leak privacy on their sources. Conversely, training such models in isolation arguably prohibits generalization. To this end, we propose a framework, NoEsis, which builds upon the desired properties of modularity, privacy, and knowledge transfer. NoEsis integrates differential privacy with a hybrid two-staged parameter-efficient fine-tuning that combines domain-specific low-rank adapters, acting as experts, with common prompt tokens, acting as a knowledge-sharing backbone. Results from our evaluation on CodeXGLUE showcase that NoEsis can achieve provable privacy guarantees with tangible knowledge transfer across domains, and empirically show protection against Membership Inference Attacks. Finally, on code completion tasks, NoEsis bridges at least 77% of the accuracy gap between the non-shared and the non-private baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。