LLM能替代文本嵌入模型吗?实测发现成本差距巨大,需按任务分工使用。
The Embedder's Dilemma: LLMs Are Better, but at What Cost?

- 对比10个LLM和26个嵌入模型在37项任务的表现,评估性能与成本
- 最优LLM与嵌入模型性能相差仅0.4分,但成本高至1431倍,推理慢最多736倍
- 适合用嵌入模型做分类聚类,用LLM处理需要推理的检索任务
是否应将文本嵌入管道替换为大语言模型?我们通过一个受控、成本意识的对比,在六个家族的10个LLM和26个嵌入模型(参数量118M到14B)上,对37项任务进行了评估,涵盖分类、语义文本相似性(STS)、聚类、成对分类和检索。总体来看,两种范式表现相当:最佳LLM(Gemini 3.1 Pro,77.6分)与最佳嵌入模型(77.2分)仅差0.4分。不同任务中优势各异:LLM在推理密集型检索中领先,嵌入模型在分类中占优,而在聚类、STS和成对分类任务上两者持平。实现这一平衡代价高昂:最贵的LLM成本是同质量嵌入模型的1431倍(154美元 vs. 0.11美元/基准测试),且开源LLM在相同GPU上处理令牌速度慢2.5至736倍。推理令牌占LLM推理成本的28%至81%;降低推理预算可维持甚至提升多数模型的检索质量。帕累托前沿包含领先的嵌入模型和一个LLM——Gemini 3.1 Pro。结果支持分工策略:用嵌入模型处理相似性、分类与聚类,保留LLM用于推理密集型检索。代码、数据集与结果已公开于https://github.com/embeddings-benchmark/embedders-dilemma。
原文摘要 · Abstract (English)
Should you replace your text-embedding pipeline with a large language model? We answer this with a controlled, cost-aware comparison of ten LLMs across six families and 26 embedding models (118M to 14B parameters) on 37 tasks spanning classification, semantic textual similarity (STS), clustering, pair classification, and retrieval. In aggregate the two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) differ by 0.4 points. Their strengths differ by task: LLMs lead on reasoning-heavy retrieval, embedding models lead on classification, and the two match on clustering, STS, and pair classification. Reaching that parity is expensive. An LLM costs up to 1,431x more than an embedding model of comparable quality (USD 154 vs. USD 0.11 per benchmark pass), and the open LLMs tested process tokens 2.5 to 736x more slowly on the same GPU. Reasoning tokens account for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improve retrieval quality for most models in our ablation. The Pareto frontier contains the leading embedding models and one LLM, Gemini 3.1 Pro. These results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval. Our code, datasets, and results are publicly available at https://github.com/embeddings-benchmark/embedders-dilemma.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。