超小模型实现顶尖多语言检索性能，800万参数比肩大模型。
Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders
- 剪枝mmBERT-small至4/13层，两阶段训练：大规模对比学习+长文档难样本微调。
- a8m仅798万非嵌入参数，却在MMTEB上达56.2分，超越40倍参数的BGE-M3。
- 模型可压缩至124MB，在浏览器和树莓派上高效运行，适合边缘部署。
多语言检索模型能做到多小？我们提出Bekko Embedding：其最小模型bekko-embedding-v1-a8m仅含约798万非嵌入参数（AP），但在官方MMTEB Multilingual v2检索任务（nDCG@10）上得分56.2，高于多语言-e5系列和BGE-M3（参数量为40倍）。更高性能的v1-a25m（约2500万AP）达57.5，与gte-multilingual-base相当。在14语言的Multilingual NanoBEIR上验证了该趋势。两者支持最长8192词输入，且在NanoLongEmbed长文本检索中表现最强。方法简单：将22层mmBERT-small剪枝至4/13层，经约11亿条多语言语料的对比学习，再用8192词长文档难负样本进行硬负样本微调，结合掩码对比损失（方向依对类型而定）与马特里什卡目标。无教师蒸馏，单卡训练约3天完成。小参数直接提升速度：相同条件下，a8m在CPU和GPU上均最快，x86 CPU上为multilingual-e5-small的1.6倍，且在Raspberry Pi 5上最快。384维输出可截断至256/128/64，便于相似性搜索与索引；词汇表嵌入行级int8量化后，a8m ONNX/OpenVINO版本仅124 MiB，可通过Transformers.js在浏览器运行。为支持可复现研究，我们发布模型权重、完整阶段一语料及独立挖掘的阶段二难负样本。
原文摘要 · Abstract (English)
How small can a competitive multilingual retrieval model be? We present Bekko Embedding: its smallest model, bekko-embedding-v1-a8m, has just under 8M Active Parameters (AP) -- the non-embedding parameters that dominate inference compute -- yet on official MMTEB Multilingual v2 Retrieval (nDCG@10) it scores 56.2, above the multilingual-e5 family and BGE-M3 (40x the AP) in our comparison. The higher-quality bekko-embedding-v1-a25m (just under 25M AP) reaches 57.5, on par with gte-multilingual-base, and Multilingual NanoBEIR (14 languages) confirms the trend. Both models handle inputs of up to 8192 tokens, and on long-input retrieval (NanoLongEmbed) a25m is the strongest dense model in our comparison. The recipe is deliberately simple. We prune the 22-layer multilingual encoder mmBERT-small to 4 / 13 layers and train the pruned models in two stages -- large-scale contrastive learning on about 1.1 billion multilingual pairs from our public corpus, followed by hard-negative fine-tuning with 8192-token long-document negatives -- with a masked contrastive loss whose direction depends on pair type, plus the Matryoshka objective. No teacher distillation is used, and all training completes on a single GPU in about 3 days for a8m. Small AP pays off directly in speed: among the compared models measured under identical conditions, a8m is the fastest on both CPU and GPU -- 1.6x multilingual-e5-small on x86 CPU -- and the fastest on a Raspberry Pi 5. The 384-dimensional output (truncatable to 256/128/64) keeps similarity search and indexing cheap, and row-wise int8 quantization of the vocabulary embedding shrinks the a8m ONNX / OpenVINO build to 124 MiB, which runs in the browser via Transformers.js. To support reproducible research, we release the model weights, the complete stage-1 corpus, and the independently mined stage-2 hard negatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。