用任务导向的蒸馏法训练出更小更强的文本嵌入模型。
jina-embeddings-v5-text: Task-Targeted Embedding Distillation
- 结合任务特定对比损失与模型蒸馏，优化小模型性能。
- 小模型在多个基准上超越或媲美当前顶尖水平。
- 支持长文本、截断和二值量化，适合部署落地。
文本嵌入模型广泛应用于语义相似性任务，如信息检索、聚类和分类。通用模型通常通过单阶段或多阶段过程，使用对比损失函数训练。我们提出一种新训练方法，将模型蒸馏与任务特定对比损失相结合，生成紧凑且高性能的嵌入模型。实验表明，该方法在训练小型模型时优于纯对比或纯蒸馏训练。所获模型 jina-embeddings-v5-text-small 与 jina-embeddings-v5-text-nano 在同类规模中表现达到或超过当前最优水平。此外，jina-embeddings-v5-text 模型支持多语言长文本（最高达32k token），在截断和二值量化下仍保持嵌入鲁棒性。模型权重已公开，有望推动嵌入模型发展。
原文摘要 · Abstract (English)
Text embedding models are widely used for semantic similarity tasks, including information retrieval, clustering, and classification. General-purpose models are typically trained with single- or multi-stage processes using contrastive loss functions. We introduce a novel training regimen that combines model distillation techniques with task-specific contrastive loss to produce compact, high-performance embedding models. Our findings suggest that this approach is more effective for training small models than purely contrastive or distillation-based training paradigms alone. Benchmark scores for the resulting models, jina-embeddings-v5-text-small and jina-embeddings-v5-text-nano, exceed or match the state-of-the-art for models of similar size. jina-embeddings-v5-text models additionally support long texts (up to 32k tokens) in many languages, and generate embeddings that remain robust under truncation and binary quantization. Model weights are publicly available, hopefully inspiring further advances in embedding model development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。