轻量级文本嵌入模型EmbeddingGemma，性能媲美双倍参数模型。
EmbeddingGemma: Powerful and Lightweight Text Representations
- 用编码器-解码器初始化+几何嵌入蒸馏，从大模型学知识。
- 300M参数在MTEB多语言/代码任务上达顶尖水平。
- 低延迟高吞吐，适合手机等设备端部署。
我们提出EmbeddingGemma，一个基于Gemma 3系列的轻量级开源文本嵌入模型。通过编码器-解码器初始化与几何嵌入蒸馏相结合的创新训练方法，有效捕获大模型知识；引入散开正则化提升模型鲁棒性与表达力，并通过融合多样化优化检查点增强泛化能力。在涵盖多语言、英语和代码领域的海量文本嵌入基准（MTEB）上，300M参数的EmbeddingGemma达到当前最佳表现。其性能超越多数已有领先模型（包括闭源和开源），且仅需不到500M参数，性能可比双倍规模模型，在量化权重或截断嵌入输出时仍保持优势。该特性使其特别适合低延迟、高吞吐场景，如设备端应用。我们提供了关键设计选择的消融研究，并向社区开源模型以推动后续研究。
原文摘要 · Abstract (English)
We introduce EmbeddingGemma, a new lightweight, open text embedding model based on the Gemma 3 language model family. Our innovative training recipe strategically captures knowledge from larger models via encoder-decoder initialization and geometric embedding distillation. We improve model robustness and expressiveness with a spread-out regularizer, and ensure generalizability by merging checkpoints from varied, optimized mixtures. Evaluated on the Massive Text Embedding Benchmark (MTEB) across multilingual, English, and code domains, EmbeddingGemma (300M) achieves state-of-the-art results. Notably, it outperforms prior top models, both proprietary and open, with fewer than 500M parameters, and provides performance comparable to models double its size, offering an exceptional performance-to-cost ratio. Remarkably, this lead persists when quantizing model weights or truncating embedding outputs. This makes EmbeddingGemma particularly well-suited for low-latency and high-throughput use cases such as on-device applications. We provide ablation studies exploring our key design choices. We release EmbeddingGemma to the community to promote further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。