提出新指标评估多语言嵌入的跨语言对齐能力，发现模型规模不决定对齐效果。
Benchmarking Cross-Lingual Semantic Alignment in Multilingual Embeddings
- 用余弦距离和PHATE可视化构建可量化对齐程度的语义亲和度（SA）指标
- 13个模型测试显示：顶流BERT类模型SA达0.70，大模型嵌入仅在0.55-0.61间波动
- 强调翻译对齐监督比模型大小或语料量更重要，适合选型参考
面对数百种多语言嵌入模型，从业者难以判断哪些真正实现跨语言语义对齐，而非仅依赖特定语言模式。任务驱动基准（如MTEB）可能掩盖根本性对齐缺陷。我们提出语义亲和度（SA），一个取值范围为0到1的有界指标，通过余弦距离衡量跨语言与同语言分布的比率，并结合PHATE可视化构建Semanscope框架。在4个数据集上对13个模型进行52次实验，揭示三层次结构：（1）顶级BERT模型（LaBSE SA=0.70，USE SA=0.68，S-BERT SA=0.68）通过翻译对监督实现强对齐；（2）大语言模型嵌入无论规模从0.6亿到80亿参数，其SA均稳定在0.55至0.61之间；（3）仅使用MLM训练的BERT模型（mBERT、XLM-R，SA<0.50）尽管训练语言超百种仍表现不佳。训练目标决定对齐效果，而非架构或规模。甲骨文原型（公元前1200年）揭示语义漂移——模型学习的是语料模式，而非认知基本单位。本研究提供语义基准，帮助从业者从众多模型中筛选高质量多语言嵌入，表明跨语言对齐需显式翻译监督，非单纯扩大模型或语料。
原文摘要 · Abstract (English)
With hundreds of multilingual embedding models available, practitioners lack clear guidance on which provide genuine cross-lingual semantic alignment versus task performance through language-specific patterns. Task-driven benchmarks (MTEB) may mask fundamental alignment shortcomings. We introduce Semantic Affinity (SA), a bounded (between 0 and 1) metric measuring inter-lingual to intra-lingual spread ratio using cosine distance, combined with PHATE visualization in our Semanscope framework. Benchmarking 13 models across 4 datasets (52 experiments) reveals a three-tier structure: (1) Top BERT models (LaBSE SA = 0.70, USE SA = 0.68, S-BERT SA = 0.68) achieve strong alignment via translation-pair supervision; (2) LLM embeddings plateau at SA between 0.55 and 0.61 regardless of 0.6 B to 8 B scale; (3) MLM-only BERT models (mBERT, XLM-R, SA < 0.50) fail despite more than 100 language training. Training objective, not architecture or scale, determines alignment. Oracle Bone primitives (1200 BCE) expose semantic drift-models learn corpus patterns rather than cognitive primitives. This work provides semantic benchmarking to help practitioners select quality multilingual embeddings from hundreds of available models, showing cross-lingual alignment requires explicit translation supervision, not merely model scale or multilingual data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。