合成型语言在BPE分词下更高效，提升语言建模效果
Morphological Typology in BPE Subword Productivity and Language Modeling
- 对比合成与分析型语言的BPE分词表现
- 合成语言子词规律性强，模型性能更好
- 适合关注分词策略与语言类型关系的研究者
本研究探讨形态类型对分词与语言建模性能的影响。聚焦具有合成与分析性形态结构的语言，考察其在字节对编码（BPE）算法下的子词生产力。在训练数据量相近的条件下比较不同语言模型的表现。实验发现，具有合成特征的语言在BPE分词中表现出更强的子词规律性与生产力，语言建模任务中取得更优结果。多个实验也验证了语言学理论中的类型连续体。这些发现表明形态类型与BPE分词效率存在相关性。
原文摘要 · Abstract (English)
This study investigates the impact of morphological typology on tokenization and language modeling performance. We focus on languages with synthetic and analytical morphological structures and examine their productivity when tokenized using the byte-pair encoding (BPE) algorithm. We compare the performance of models trained with similar amounts of data in different languages. Our experiments reveal that languages with synthetic features exhibit greater subword regularity and productivity with BPE tokenization and achieve better results in language modeling tasks. We also observe that the typological continuum from linguistic theory is reflected in several experiments. These findings suggest a correlation between morphological typology and BPE tokenization efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。