自适应加噪生成合成数据，保护隐私同时保持大模型上下文学习精度。
Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning
- 根据数据统计特性动态调整加噪强度，优化隐私与性能平衡。
- 在标准基准上接近非私有基线的准确率，优于现有差分隐私方法。
- 适合需要隐私保护的提示工程场景，如医疗、金融等敏感领域。
大型语言模型依赖提示中的示例/演示所包含的上下文信息进行上下文学习（ICL）。为降低模型在提示中泄露私有数据的风险，本文提出一种新型数据自适应差分隐私算法AdaDPSyn，从私有数据集中生成合成示例，并用其执行ICL。AdaDPSyn的目标是根据数据内在统计特性自适应调整数据合成机制中的噪声水平，从而在保持正式差分隐私保障的同时，维持高ICL准确率。其核心创新在于精度导向的迭代半径缩减技术，该技术基于数据聚类模式动态优化聚合半径（即噪声添加的数据分组范围），以最小化所需加噪量。我们在标准基准上进行了大量实验，将AdaDPSyn与差分隐私少样本生成算法（Tang et al., 2023）对比，结果表明：AdaDPSyn不仅优于该基线方法，且准确率接近非私有基线水平，为具有隐私保护的ICL提供了有效解决方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) rely on the contextual information embedded in examples/demonstrations to perform in-context learning (ICL). To mitigate the risk of LLMs potentially leaking private information contained in examples in the prompt, we introduce a novel data-adaptive differentially private algorithm called AdaDPSyn to generate synthetic examples from the private dataset and then use these synthetic examples to perform ICL. The objective of AdaDPSyn is to adaptively adjust the noise level in the data synthesis mechanism according to the inherent statistical properties of the data, thereby preserving high ICL accuracy while maintaining formal differential privacy guarantees. A key innovation in AdaDPSyn is the Precision-Focused Iterative Radius Reduction technique, which dynamically refines the aggregation radius - the scope of data grouping for noise addition - based on patterns observed in data clustering, thereby minimizing the amount of additive noise. We conduct extensive experiments on standard benchmarks and compare AdaDPSyn with DP few-shot generation algorithm (Tang et al., 2023). The experiments demonstrate that AdaDPSyn not only outperforms DP few-shot generation, but also maintains high accuracy levels close to those of non-private baselines, providing an effective solution for ICL with privacy protection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。