Giga-Embeddings用专家混合模型实现高速文本嵌入，兼顾性能与效率。
Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
- 采用稀疏专家混合架构，每令牌仅激活约18亿参数。
- 在多语言和代码MTEB基准上表现最优，1024 token输入下吞吐达11.45万tokens/秒。
- 提供小规模版本，480M模型参数少42%却超越FRIDA，适合资源受限场景。
我们提出Giga-Embeddings，一类兼具强检索性能与高效服务能力的文本嵌入模型。其最大版本为稀疏的100亿参数专家混合编码器，每令牌激活约18亿参数。在英文、俄文、多语言及代码的MTEB基准上，该模型在所有四个评测套件中均取得家族内最佳综合表现。在vLLM基准中，1024词元输入下处理速度达11.45万词元/秒，较稠密30亿参数模型提升25%，是所测外部系统1.56至2.65倍。家族还包括一个稠密30亿参数编码器和一个蒸馏后的4.8亿参数编码器，适用于更紧的计算与内存预算。紧凑模型通过无维度目标训练，使师生相似性分布对齐。最终480M模型在俄文MTEB上得分为70.98，超越FRIDA且参数减少42%。所有三个模型检查点均已公开。
原文摘要 · Abstract (English)
We introduce Giga-Embeddings, a family of text embedding models designed to combine strong retrieval quality with efficient serving. Its largest member is a sparse 10B-parameter Mixture-of-Experts encoder with approximately 1.8B active parameters per token. Across English, Russian, multilingual, and code MTEB benchmarks, this model achieves the strongest aggregate performance within the family on all four evaluated suites. In our vLLM benchmark with 1024-token inputs, it processes 114.5k tokens per second, providing 25 percent higher throughput than the dense 3B model and 1.56-2.65x the throughput of the evaluated external systems. The family also includes a dense 3B encoder and a distilled 480M encoder for tighter compute and memory budgets. We train the compact model using a dimension-agnostic objective that aligns teacher and student similarity distributions. The resulting 480M model scores 70.98 on Russian MTEB, surpassing FRIDA while using 42 percent fewer parameters. We release all three model checkpoints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。