用子词方法分析242种语言的词汇差异，揭示语言亲缘关系与分词规律。
Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets
- 基于维基百科构建语言语料集，用字节对编码分析跨语言词汇相似性。
- 95%的分词准确率优于随机基线，语言间相似度与谱系关系显著相关。
- 适合语言学、计算语言学研究者，探索跨语言词汇演化模式。
我们利用子词方法对242种拉丁与西里尔字母语言开展大规模比较研究。通过从维基百科词典构建‘语系集’，引入基于字节对编码（BPE）的统一框架，实现多语言同步对比。采用基于排名的子词向量分析词汇重叠、词法差异与语言相似性。评估显示，BPE分词在15种语言上比随机基线高出95%（F1=0.34 vs 0.15），且词表相似性与语言谱系关系显著相关（Mantel r = 0.329, p < 0.001）。罗曼语族形成最紧密聚类（平均距离0.51），跨语系配对清晰分离（0.82）。对26,939个跨语言同形词分析发现，48.7%在相关语言中分词不同，其差异程度与谱系距离相关。结果为类型多样语言提供了统一的宏观语言学量化洞察。
原文摘要 · Abstract (English)
We present a large-scale comparative study of 242 Latin and Cyrillic-script languages using subword-based methodologies. By constructing 'glottosets' from Wikipedia lexicons, we introduce a framework for simultaneous cross-linguistic comparison via Byte-Pair Encoding (BPE). Our approach utilizes rank-based subword vectors to analyze vocabulary overlap, lexical divergence, and language similarity at scale. Evaluations demonstrate that BPE segmentation aligns with morpheme boundaries 95% better than random baseline across 15 languages (F1 = 0.34 vs 0.15). BPE vocabulary similarity correlates significantly with genetic language relatedness (Mantel r = 0.329, p < 0.001), with Romance languages forming the tightest cluster (mean distance 0.51) and cross-family pairs showing clear separation (0.82). Analysis of 26,939 cross-linguistic homographs reveals that 48.7% receive different segmentations across related languages, with variation correlating to phylogenetic distance. Our results provide quantitative macro-linguistic insights into lexical patterns across typologically diverse languages within a unified analytical framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。