对比三种分词法，发现基于词素的分词在库尔德语词向量中表现更优。
Subword Tokenization Strategies for Kurdish Word Embeddings
- 用自举训练构建词素分割器，比较词级、词素级和BPE分词法
- 词素法覆盖率达68.7%，远超BPE的28.6%，且语义结构更优
- 强调低资源语言评估需考虑覆盖率，避免结果失真
我们通过比较词级、基于词素和BPE三种分词策略，在形态相似性保持任务中研究库尔德语词向量的表示效果。采用自举训练从少量人工标注数据构建BiLSTM-CRF词素分割器，并使用Word2Vec在相似性保持、聚类质量及语义组织等多个指标上评估嵌入性能。分析显示，传统对比存在严重评估偏差：尽管BPE在形态相似性上看似更优，但仅能覆盖28.6%的测试样本，而词素模型覆盖率达68.7%，导致其性能被人为夸大。全面评估表明，词素分词法在嵌入空间组织、语义邻域结构及不同复杂度层级的平衡覆盖方面均表现更佳。该研究强调了在低资源语言处理中进行覆盖率感知评估的重要性，并为低资源语言提供了多样化的分词方法。
原文摘要 · Abstract (English)
We investigate tokenization strategies for Kurdish word embeddings by comparing word-level, morpheme-based, and BPE approaches on morphological similarity preservation tasks. We develop a BiLSTM-CRF morphological segmenter using bootstrapped training from minimal manual annotation and evaluate Word2Vec embeddings across comprehensive metrics including similarity preservation, clustering quality, and semantic organization. Our analysis reveals critical evaluation biases in tokenization comparison. While BPE initially appears superior in morphological similarity, it evaluates only 28.6\% of test cases compared to 68.7\% for morpheme model, creating artificial performance inflation. When assessed comprehensively, morpheme-based tokenization demonstrates superior embedding space organization, better semantic neighborhood structure, and more balanced coverage across morphological complexity levels. These findings highlight the importance of coverage-aware evaluation in low-resource language processing and offers different tokenization methods for low-resourced language processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。