通过加权多语言模型融合，实现高保真私有数据合成且满足差分隐私。
Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion
- 用投票机制结合多个预训练模型，更准估计私有数据分布。
- 在6个数据集上提升下游任务性能，优于现有方法。
- 适合数据少、隐私要求高的场景，如医疗或金融建模。
高质量与充足数量是构建优质训练数据集的黄金标准，同时保护样本隐私同样重要。通过生成与高质量私有数据相似且满足差分隐私（DP）的合成样本，可实现可扩展性和实用性。然而，现有依赖预训练模型进行数据合成的方法（避免微调大型生成模型）在数据稀缺场景下表现不佳，存在样本量有限、生成噪声明显以及预训练模型固有偏差等问题。为此，我们提出一种基于加权多预训练语言模型（PLM）融合的对比私有数据合成框架——WASP。WASP 利用少量私有样本通过 Top-Q 投票机制更准确地估计私有数据分布，并借助低质量合成样本，通过动态加权的多个 PLM 协作实现对比生成。在 6 个成熟数据集上，使用 6 个开源和 3 个闭源 PLM 的大量实验表明，WASP 在多样化下游任务中显著提升模型性能。代码已公开于 https://anonymous.4open.science/r/WASP。
原文摘要 · Abstract (English)
Substantial quantity and high quality are the golden rules of making a good training dataset with sample privacy protection equally important. Generating synthetic samples that resemble high-quality private data while ensuring Differential Privacy (DP), a formal privacy guarantee, promises scalability and practicality. However, existing methods relying on pre-trained models for data synthesis %that avoid fine-tuning large pre-trained generative models often struggle in data-deficient scenarios, suffering from limited sample size, inevitable generation noise and existing pre-trained model bias. To address these challenges, we propose a novel contrAstive private data Synthesis via Weighted multiple Pre-trained language models (PLM) framework, named as WASP. WASP utilizes limited private samples for more accurate private data distribution estimation via a Top-Q voting mechanism, and leverages low-quality synthetic samples for contrastive generation via collaboration among dynamically weighted multiple pre-trained models.Extensive experiments on 6 well-developed datasets with 6 open-source and 3 closed-source PLMs demonstrate the superiority of WASP in improving model performance over diverse downstream tasks. Code is available at https://anonymous.4open.science/r/WASP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。