MAPLE让私有语言模型微调更高效,解决初始化难题
MAPLE: Metadata Augmented Private Language Evolution
- 用元数据和上下文学习初始化合成数据,精准匹配目标领域
- 相比基线方法,隐私-效用权衡更优,收敛速度提升显著
- 适合需隐私保护的领域文本生成,降低API调用成本
差分隐私微调大语言模型需要大量算力和全模型访问权限,使普通用户无法使用主流专有API。生成差分隐私合成数据是可行替代方案,支持透明的数据分析和跨任务复用,规避模型参数空间的刚性限制。私有演化(PE)提供基于API的生成框架,但其效果高度依赖初始化。当私有数据分布偏离基础模型预训练先验时——在高度专业化领域中常见——PE难以对齐目标数据,导致收敛差、效用下降和API浪费。为此,我们提出元数据增强的私有语言演化(MAPLE)。MAPLE提取差分隐私的表格元数据,并利用上下文学习将初始合成分布牢固锚定于目标领域。在特定领域文本生成任务上的评估表明,MAPLE在隐私-效用权衡上表现更优,收敛更快,显著降低API开销。
原文摘要 · Abstract (English)
Differentially private (DP) fine-tuning of large language models (LLMs) requires massive compute and full model access, which rules out state-of-the-art proprietary APIs for general users. Generating DP synthetic data offers a practical workaround. This approach also allows for transparent exploratory data analysis and arbitrary reuse across downstream tasks, sidestepping the rigid constraints of a model's parameter space. Private Evolution (PE) provides a promising API-based framework for generating this data, but its success relies heavily on initialization. If the private data distribution falls too far outside the foundation model's pre-training priors -- a common issue in highly specialized domain -- PE struggles to align with the target data. This misalignment causes poor convergence, degraded utility, and wasted API calls. To solve this initialization bottleneck, we introduce Metadata Augmented Private Language Evolution (MAPLE). MAPLE extracts DP tabular metadata and uses in-context learning to firmly ground the initial synthetic distribution in the target domain. Our evaluations on domain-specific text generation tasks show that MAPLE yields a strictly better privacy-utility trade-off, converges significantly faster, and sharply reduces API costs compared to baseline PE methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。