打造跨语言跨模态统一语义空间,支持数千语言文本与语音嵌入。
Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech
- 用渐进式训练构建基础语义空间,再通过师生蒸馏扩展至数千语言
- 在200语言FLORES上相似度搜索误差减半,在1560语言BIBLE上降15倍
- 零样本语音翻译达SeamlessM4T97%性能,适合多语言任务研究者
跨语言句子编码器通常仅覆盖数百种语言,且常以牺牲下游性能换取更强对齐,限制了实际应用。本文提出OmniSONAR,一种全新的全语言、跨语言、跨模态句子嵌入模型家族,可原生将文本、语音、代码和数学表达嵌入同一语义空间,并在涵盖数千种语言(从高资源到极低资源)的规模上实现最先进的下游性能。为避免表示坍缩,采用渐进训练策略:首先使用大语言模型初始化的编码器-解码器,在200种语言上学习强健的基础空间,结合词元级解码与新颖的分软最大对比损失及合成硬负例;在此基础上,通过两阶段师生编码器蒸馏框架扩展至数千语言变体;最后,无缝将177种口语语言映射至该空间。OmniSONAR在200语言FLORES数据集上将跨语言相似度搜索误差减半,在1560语言BIBLE基准上降低15倍。它在多语言翻译中表现优异,超越NLLB-3B并在1560语言转英文圣经任务上超过先前模型(包括更大规模的LLM)15 chrF++点。在MTEB和XLCoST上也表现强劲。对于语音,其相似度搜索误差降低43%,尽管零样本翻译(仅在语音识别数据上训练),仍达到SeamlessM4T语音转文本质量的97%。此外,通过仅在英语文本处理任务上训练基于OmniSONAR嵌入序列的编码器-解码器语言模型Spectrum,成功实现对数千语言及语音的高性能迁移,适用于复杂下游任务。
原文摘要 · Abstract (English)
Cross-lingual sentence encoders typically cover only a few hundred languages and often trade downstream quality for stronger alignment, limiting their adoption. We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-modal sentence embedding models that natively embed text, speech, code, and mathematical expressions in a single semantic space, while delivering state-of-the-art downstream performance at the scale of thousands of languages, from high-resource to extremely low-resource varieties. To reach this scale without representation collapse, we use progressive training. We first learn a strong foundational space for 200 languages with an LLM-initialized encoder-decoder, combining token-level decoding with a novel split-softmax contrastive loss and synthetic hard negatives. Building on this foundation, we expand to several thousands language varieties via a two-stage teacher-student encoder distillation framework. Finally, we demonstrate the cross-modal extensibility of this space by seamlessly mapping 177 spoken languages into it. OmniSONAR halves cross-lingual similarity search error on the 200-language FLORES dataset and reduces error by a factor of 15 on the 1,560-language BIBLE benchmark. It also enables strong translation, outperforming NLLB-3B on multilingual benchmarks and exceeding prior models (including much larger LLMs) by 15 chrF++ points on 1,560 languages into English BIBLE translation. OmniSONAR also performs strongly on MTEB and XLCoST. For speech, OmniSONAR achieves a 43% lower similarity-search error and reaches 97% of SeamlessM4T speech-to-text quality, despite being zero-shot for translation (trained only on ASR data). Finally, by training an encoder-decoder LM, Spectrum, exclusively on English text processing OmniSONAR embedding sequences, we unlock high-performance transfer to thousands of languages and speech for complex downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。