首个斯洛伐克语文本嵌入基准，助力低资源语言语义理解
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation
- 构建涵盖31个数据集的斯洛伐克语嵌入评测基准，任务类型达7类
- 自研小型模型在保留62%参数下仍媲美商业API，支持本地部署
- 为低资源语言提供可复现的嵌入模型构建路径，适合本地化应用
我们提出SkMTEB，首个面向斯洛伐克语（一种低资源西斯拉夫语言）的综合性文本嵌入基准，包含31个数据集，覆盖7种任务类型，规模接近现有多语言基准的4倍。对31个嵌入模型的评估显示，大型指令微调的多语言模型表现最佳，而现有针对自然语言理解任务训练的斯洛伐克专用模型在嵌入任务上迁移效果差。为满足高效、本地部署的需求，我们通过词汇剪枝与微调，将多语言E5模型改进为开源的\texttt{e5-sk-small}（45M参数）和\texttt{e5-sk-large}（365M）。尽管参数量减少最高达62%，其性能仍可媲美商业API，适用于语义搜索与检索增强生成（RAG）。我们开放发布基准、模型、数据集与代码，旨在为其他低资源语言提供可复现的嵌入模型构建路径。
原文摘要 · Abstract (English)
We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types -- nearly 4$\times$ the depth of existing multilingual benchmark coverage for Slovak. Our evaluation of 31 embedding models reveals that large instruction-tuned multilingual models achieve the strongest performance, while existing Slovak-specific models trained for NLU tasks transfer poorly to embedding tasks. To address the need for efficient, locally-deployable Slovak embeddings, we develop \texttt{e5-sk-small} (45M parameters) and \texttt{e5-sk-large} (365M) by applying vocabulary trimming and fine-tuning to Multilingual E5 models. Despite size reductions of up to 62\%, our open-source models achieve competitive performance with proprietary APIs while remaining locally deployable for semantic search and retrieval-augmented generation (RAG). We release the benchmark, models, datasets, and code openly, hoping our approach offers a replicable path for other under-resourced languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。