用多语言大模型嵌入向量自动发现跨语言词汇空缺
Discovering Lexical Gaps Using Embeddings from Multilingual LLMs
- 通过多语言模型生成嵌入空间，计算源语言词与目标语言近邻的语义相似度
- 94%~97%的嵌入空间中，词汇空缺词的跨语言对齐更弱，分类器准确率达0.81以上
- 无需人工标注或概念分类，适合大规模跨语言资源建设
词汇空缺指某些语言中不存在的词汇，影响多语言词典构建、机器翻译和跨语言迁移。现有方法依赖人工判断或固定概念体系。本文提出一种数据驱动框架，从韩英双语大模型中提取上下文嵌入，针对韩译英和英译韩翻译对构建了每种语言4000个不同的嵌入空间（基于100次训练-测试划分）。在每个空间中，计算源语言词与其目标语言最近邻的语义相似度，并比较空缺词与非空缺词的分布差异。在94%（韩→英）和97%（英→韩）的嵌入空间中，空缺词的跨语言语义对齐明显弱于非空缺词。基于未对齐嵌入空间训练的逻辑回归分类器可可靠区分空缺词，韩→英的AUC达0.81，英→韩为0.76，并成功召回18/19个韩语空缺词和26/27个英语空缺词。该方法实现无语言依赖、无分类体系的可扩展词汇空缺识别。
原文摘要 · Abstract (English)
Lexical gaps are words that do not exist in certain languages. They pose challenges for building multilingual lexical resources, for machine translation, and for cross-lingual transfer. Existing lexical gap detection relies on human judgments or fixed conceptual taxonomies. We propose a data-driven framework for identifying cross-lingual lexical gaps. We extracted contextualized embeddings from Korean-English bilingual LLMs for Korean-to-English and English-to-Korean translation pairs. Combinations of LLMs, embedding types, dimensionality, and orthogonal transformations across 100 train-test splits yielded 4000 distinct embedding spaces in each source language. In each space, we computed the semantic similarity between each source word and its nearest neighbor in the target language, and compared their distribution for gap words versus non-gap words. In 94% (Korean-to-English) and 97% (English-to-Korean) of embedding spaces, gap words showed weaker cross-lingual semantic alignment than non-gap words. Logistic classifiers trained on unaligned embedding spaces can reliably separate gap words from non-gap words, achieving AUCs of 0.81 (Korean-to-English) and 0.76 (English-to-Korean) and retrieving 18/19 Korean and 26/27 English gap words. This approach provides a language-agnostic and taxonomy-free method for scalable lexical gap identification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。