用数据向量高效私密生成合成数据,低资源下仍保持高保真。
EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectors
- 通过数据向量捕捉私有数据与公开先验的分布差异,实现轻量级隐私保护。
- 仅需一次向量提取即可生成任意数量样本,隐私成本不随生成量增加。
- 适用于小样本场景,适合对数据隐私和计算效率要求高的应用。
高质量数据对现代机器学习至关重要,但许多有价值的数据集具有敏感性,无法自由共享。合成数据为下游开发提供了实用替代方案,大语言模型(LLMs)已成为生成合成数据的强大引擎。然而,现有私密文本生成方法效率极低:依赖大量数据、计算缓慢,且通常需要大规模私有语料或批量处理才能获得可用质量。我们提出EPSVec,一种差分隐私下的轻量级替代方案,通过*数据集向量*——激活空间中反映私有数据与公共先验分布差异的方向——引导LLM生成。EPSVec仅需一次性提取并净化方向向量,随后进行标准解码。该方法将隐私预算与生成过程解耦,可无额外隐私成本生成任意数量合成样本,在低数据条件下仍保持强保真度。此外,我们通过使用预训练(基础)模型并引入固定提示(fixed-shot prompting),进一步提升生成多样性与保真度。实验表明,与现有基线相比,EPSVec在分布对齐和下游任务效用方面表现更优,尤其在低数据场景下,同时显著降低计算开销。
原文摘要 · Abstract (English)
High-quality data is essential for modern machine learning, yet many valuable corpora are sensitive and cannot be freely shared. Synthetic data offers a practical substitute for downstream development, and large language models (LLMs) have emerged as powerful engines for generating it. However, existing private text generation methods are severely inefficient: they are data-intensive, computationally slow, and often require large private corpora or batch sizes to achieve usable quality. We introduce EPSVec, a differentially-private lightweight alternative that steers LLM generation using *dataset vectors*--directions in activation space that capture the distributional gap between private data and public priors. EPSVec extracts and sanitizes steering vectors just once and then performs standard decoding. This decouples the privacy budget from generation, enabling arbitrarily many synthetic samples without additional privacy cost and yielding strong fidelity even in low-data regimes. Furthermore, we enhance our method by utilizing pretrained (base) models and introducing fixed-shot prompting to boost generation diversity and fidelity. Our experiments demonstrate that EPSVec outperforms existing baselines in distributional alignment and downstream utility, particularly in low-data regimes, while significantly reducing computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。