用梯度匹配生成可读合成文本，训练大模型更高效且保隐私。
Synthetic Text Generation for Training Large Language Models via Gradient Matching
- 通过ADMM优化合成数据嵌入，匹配真实数据梯度。
- 合成文本使模型收敛到真实微调结果附近，准确率接近真实数据训练。
- 适合需隐私保护的LLM微调场景，如医疗、金融领域。
合成数据有望提升真实训练数据的性能、训练效率和隐私保护。然而，现有合成文本生成方法多依赖启发式策略，难以在不泄露真实数据隐私的前提下生成人类可读文本，也无法为大语言模型（LLM）训练提供性能保证。本文提出首个理论严谨的合成人类可读文本生成方法，可在目标任务上对LLM微调提供收敛性、性能与隐私三重保障。我们采用交替方向乘子法（ADMM），迭代优化合成样本的嵌入，使其匹配目标训练或验证数据的噪声梯度，并将嵌入映射为低困惑度的文本序列。该方法确保模型收敛至真实数据微调解的邻域，同时保护数据隐私。在多种分类任务上的实验验证了其有效性。代码已开源：https://github.com/BigML-CS-UCLA/GRADMM。
原文摘要 · Abstract (English)
Synthetic data has the potential to improve the performance, training efficiency, and privacy of real training examples. Nevertheless, existing approaches for synthetic text generation are mostly heuristics and cannot generate human-readable text without compromising the privacy of real data, or provide performance guarantees for training Large Language Models (LLMs). In this work, we propose the first theoretically rigorous approach for generating synthetic human-readable text that provides convergence, performance, and privacy guarantees for fine-tuning LLMs on a target task. To do so, we leverage Alternating Direction Method of Multipliers (ADMM) that iteratively optimizes the embeddings of synthetic examples to match the noisy gradient of the target training or validation data, and maps them to a sequence of text tokens with low perplexity. In doing so, the generated synthetic text guarantees convergence of the model to a close neighborhood of the solution obtained by fine-tuning on real data and preserves their privacy. Experiments on various classification tasks confirm the effectiveness of our proposed approach. Our code is available at https://github.com/BigML-CS-UCLA/GRADMM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。