用600万开源数据微调，小模型也能达到顶尖嵌入效果。
F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Data
- 从基础模型直接微调，仅需600万真实数据对
- F2LLM-4B在40亿参数中排名第二，17亿参数版第一
- 开源模型、数据与代码，低成本可复现
我们提出F2LLM——一种基于三类规模（0.6B、1.7B、4B）的先进嵌入模型。与以往依赖大规模对比预训练、复杂训练流程和昂贵合成数据的顶级模型不同，F2LLM直接在600万条从开源非合成数据中筛选的查询-文档-负样本三元组上进行微调，实现了训练成本、模型规模与嵌入性能之间的良好平衡。在MTEB英文排行榜上，F2LLM-4B在约40亿参数模型中位列第2，总体排名第7；F2LLM-1.7B在10亿至20亿参数范围内排名第一。为推动该领域研究，我们开源了模型、训练数据集及代码，使F2LLM成为未来工作的一个强大、可复现且经济高效的基准。
原文摘要 · Abstract (English)
We introduce F2LLM - Foundation to Feature Large Language Models, a suite of state-of-the-art embedding models in three sizes: 0.6B, 1.7B, and 4B. Unlike previous top-ranking embedding models that require massive contrastive pretraining, sophisticated training pipelines, and costly synthetic training data, F2LLM is directly finetuned from foundation models on 6 million query-document-negative tuples curated from open-source, non-synthetic datasets, striking a strong balance between training cost, model size, and embedding performance. On the MTEB English leaderboard, F2LLM-4B ranks 2nd among models with approximately 4B parameters and 7th overall, while F2LLM-1.7B ranks 1st among models in the 1B-2B size range. To facilitate future research in the field, we release the models, training dataset, and code, positioning F2LLM as a strong, reproducible, and budget-friendly baseline for future works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。