用80亿参数开源模型高效生成高质量文本嵌入数据
Little Giants: Synthesizing High-Quality Embedding Data at Scale
- 通过微调与自我优化，让小模型生成媲美大模型的合成数据
- 仅需不到十分之一的GPT调用次数，效果超越E5_mistral
- 揭示合成数据质量的可扩展规律，适合大规模训练场景
合成数据生成已成为无需大量人工标注数据训练模型的流行方式。对于文本嵌入任务，合成数据能提供多样且可扩展的训练样本，显著降低人工标注成本。然而，当前多数方法严重依赖如GPT-4等专有模型，存在成本高、效率低的问题。本文提出SPEED框架，将80亿参数的开源小模型对齐，以高效生成大规模合成嵌入数据。通过监督微调、偏好优化和自提升机制，SPEED使小模型生成高质量数据。惊人的是，SPEED使用不到1/10的GPT API调用次数，其生成的数据训练出的模型在性能上优于仅使用合成数据训练的E5_mistral。利用该高效生成器，我们系统研究了对齐流程中各类因素对数据质量的影响，并揭示了合成嵌入数据的可扩展规律。
原文摘要 · Abstract (English)
Synthetic data generation has become an increasingly popular way of training models without the need for large, manually labeled datasets. For tasks like text embedding, synthetic data offers diverse and scalable training examples, significantly reducing the cost of human annotation. However, most current approaches rely heavily on proprietary models like GPT-4, which are expensive and inefficient for generating large-scale embedding data. In this paper, we introduce SPEED, a framework that aligns open-source small models (8B) to efficiently generate large-scale synthetic embedding data. Through supervised fine-tuning, preference optimization, and self-improvement, SPEED enables small open-source models to produce high-quality data. Remarkably, SPEED uses only less than 1/10 of the GPT API calls, outperforming the state-of-the-art embedding model E5_mistral when both are trained solely on their synthetic data. Using this efficient generator, we conduct a comprehensive study on how various factors within the alignment pipeline impact data quality and reveal the scaling law for synthetic embedding data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。