用奖励机制生成更干净的私密数据，提升大模型微调效果
RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis
- 通过奖励模型引导合成数据生成，精准过滤噪声
- 在医疗、金融等领域实现更高精度的微调性能
- 适合需保护隐私又想高效微调大模型的研究者
大语言模型的成功促使越来越多用户上传私有数据以进行领域特定任务的微调。然而，在医疗、金融等敏感领域，隐私问题尤为突出。一种有前景的解决方案是使用差分隐私（DP）保证生成合成数据以替代原始数据。但这类合成数据常包含大量错误内容，被视为噪声。现有方法通常依赖ROUGE-L或嵌入相似性进行简单筛选，效果有限。为此，我们提出RewardDS，一种新型隐私保护微调框架，通过训练奖励代理模型并利用其奖励信号指导合成数据生成。RewardDS引入两个核心模块：奖励引导过滤与自优化精炼，有效减少噪声。在医疗、金融和代码生成领域的大量实验表明，该方法显著提升了微调效果。
原文摘要 · Abstract (English)
The success of large language models (LLMs) has attracted many individuals to fine-tune them for domain-specific tasks by uploading their data. However, in sensitive areas like healthcare and finance, privacy concerns often arise. One promising solution is to generate synthetic data with Differential Privacy (DP) guarantees to replace private data. However, these synthetic data contain significant flawed data, which are considered as noise. Existing solutions typically rely on naive filtering by comparing ROUGE-L scores or embedding similarities, which are ineffective in addressing the noise. To address this issue, we propose \textit{RewardDS}, a novel privacy-preserving framework that fine-tunes a reward proxy model and uses reward signals to guide the synthetic data generation. Our \textit{RewardDS} introduces two key modules, Reward Guided Filtering and Self-Optimizing Refinement, to both filter and refine the synthetic data, effectively mitigating the noise. Extensive experiments across medical, financial, and code generation domains demonstrate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。