用轻量方法打造高性能土耳其语句子嵌入模型,成本低效果好。
Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation

- 通过优化多语言分词器+离线知识蒸馏,低成本适配土耳其语
- 在26项任务上达63.9%平均得分,参数比教师模型少33%
- 仅需单卡4小时训练,总成本5-20美元,适合实际部署
句向量是语义搜索、聚类、分类和检索增强生成的基础组件。本文提出embeddingmagibu-200m,一个专为土耳其语设计的句向量模型,输出768维L2归一化向量,支持8192个标记的上下文窗口,远超早期基于BERT的土耳其语编码器512标记限制。采用高效的三阶段适配流程:(1) 从40种语言语料中分析频率,对教师模型词汇表进行剪枝并加入多语言标记,构建131,072个标记的土耳其语优化分词器;(2) 克隆教师模型结构,保留Transformer主干权重,并通过均值组合映射初始化新词汇表的嵌入层;(3) 在平衡的40语言维基百科语料上,使用余弦相似度目标对预计算的教师向量进行离线知识蒸馏。最终学生模型约含200M参数,在单张GPU上训练耗时约4小时,避免在线教师推理,总成本仅$5-$20。实证结果表明,在STSbTR数据集上皮尔逊/斯皮尔曼相关系数达77.55%/77.45%,超越3亿参数教师模型(73.84%/72.92%)。在TR-MTEB(26项任务)上取得63.9%的平均得分(26个模型中第7名),以33%更少参数实现竞争力强的成本-质量平衡。为促进复现与下游应用,所有成果均已开源,包括模型权重、分词器文件、预计算嵌入数据集及克隆与蒸馏工具链。
原文摘要 · Abstract (English)
Sentence embeddings are a foundational component for semantic search, clustering, classification, and retrieval-augmented generation. This paper presents embeddingmagibu-200m, a Turkish-focused sentence embedding model that produces 768-dimensional L2-normalized vectors and supports an 8,192-token context window, far exceeding the 512-token limit of earlier BERT-based Turkish encoders. Instead of full pretraining, an efficient three-stage adaptation pipeline is introduced: (1) construct a Turkish-optimized multilingual tokenizer with a 131,072 vocabulary by pruning redundant tokens from the teacher's vocabulary and incorporating multilingual tokens via frequency analysis on a 40-language corpus, (2) clone a teacher embedding model while preserving transformer backbone weights and initializing a compatible embedding table for the new vocabulary via mean-composition token mapping, and (3) perform offline embedding distillation from precomputed teacher vectors using a cosine similarity objective over a balanced 40-language Wikipedia corpus. The resulting student model contains approximately 200M parameters and trains in roughly four hours on a single GPU by avoiding online teacher inference during training, at a total cost of $5-$20. Empirically, Pearson/Spearman correlations of 77.55%/77.45% are obtained on STSbTR, surpassing the 300M-parameter teacher model (73.84%/72.92%). On TR-MTEB (26 tasks), a mean score of 63.9% is achieved (7th out of 26 models), providing a competitive cost-quality trade-off with 33% fewer parameters than the teacher. To facilitate reproducibility and downstream use, all artifacts are released including model weights, tokenizer files, precomputed embedding datasets, and open-source cloning and distillation tooling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。