基于大模型生成高质量文本数据,提升文本嵌入模型的检索与泛化能力。
QZhou-Embedding Technical Report
- 用大模型生成多样化文本数据,增强训练集语义丰富性。
- 在MTEB和CMTEB双榜单排名第一,支持重排序、聚类等多任务。
- 开源权重与代码,适合需要高精度嵌入的科研与工程应用。
我们提出QZhou-Embedding,一种通用上下文文本嵌入模型,具备卓越的文本表征能力。基于Qwen2.5-7B-Instruct模型,设计统一多任务框架,包含专用数据转换与训练策略。数据转换方案引入更多样化的文本训练数据,任务特定训练策略提升学习效率。开发了基于LLM API的数据合成流水线,结合改写、增强与难例生成技术,提升训练集的语义丰富度与样本难度。采用两阶段训练策略:先进行聚焦检索的预训练,再进行全任务微调,使模型在强大检索基础上扩展能力。模型在MTEB和CMTEB基准上均取得领先成绩(截至2025年8月27日),并同时在重排序、聚类等任务中达到当前最优水平。研究证明,更高质量、更多样化的数据对提升检索模型性能至关重要,利用大模型生成能力可进一步优化数据质量,推动嵌入模型突破。模型权重已在HuggingFace以Apache 2.0许可开源。为保障可复现性,评估代码与使用说明已发布于GitHub。
原文摘要 · Abstract (English)
We present QZhou-Embedding, a general-purpose contextual text embedding model with exceptional text representation capabilities. Built upon the Qwen2.5-7B-Instruct foundation model, we designed a unified multi-task framework comprising specialized data transformation and training strategies. The data transformation scheme enables the incorporation of more diverse textual training datasets, while the task-specific training strategies enhance model learning efficiency. We developed a data synthesis pipeline leveraging LLM API, incorporating techniques such as paraphrasing, augmentation, and hard negative example generation to improve the semantic richness and sample difficulty of the training set. Additionally, we employ a two-stage training strategy, comprising initial retrieval-focused pretraining followed by full-task fine-tuning, enabling the embedding model to extend its capabilities based on robust retrieval performance. Our model achieves state-of-the-art results on the MTEB and CMTEB benchmarks, ranking first on both leaderboards (August 27 2025), and simultaneously achieves state-of-the-art performance on tasks including reranking, clustering, etc. Our findings demonstrate that higher-quality, more diverse data is crucial for advancing retrieval model performance, and that leveraging LLMs generative capabilities can further optimize data quality for embedding model breakthroughs. Our model weights are released on HuggingFace under Apache 2.0 license. For reproducibility, we provide evaluation code and instructions on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。