低资源突厥语芬兰语中，词级分词比子词更好用。
Tokenization Strategies for Low-Resource Agglutinative Languages in Word2Vec: Case Study on Turkish and Finnish
- 用词级、字符级、n-gram和BPE四种分词法训练Word2Vec
- 词级分词在命名实体识别任务上表现最佳，准确率最高
- 适合数据少、算力有限的低资源语言项目使用
分词在屈折语言中至关重要，单个词可包含多个承载句法与语义信息的词素。本研究评估了词级、字符级、n-gram及字节对编码（BPE）等分词策略对土耳其语和芬兰语静态词向量质量的影响。基于10,000篇维基百科文章的语料库，在低资源条件下训练Word2Vec模型，并在命名实体识别（NER）任务上进行评估。尽管子词分割理论上更优，词级分词在所有测试策略中均表现最佳。结果表明，在低资源屈折语言场景下，保留词边界可能比复杂统计方法产生更优的嵌入效果。这对计算资源和标注数据有限的少数语言自然语言处理流程开发具有实际意义。
原文摘要 · Abstract (English)
Tokenization plays a critical role in processing agglutinative languages, where a single word can encode multiple morphemes carrying syntactic and semantic information. This study evaluates the impact of various tokenization strategies - word-level, character-level, n-gram, and Byte Pair Encoding (BPE) - on the quality of static word embeddings generated by Word2Vec for Turkish and Finnish. Using a 10,000-article Wikipedia corpus, we trained models under low-resource conditions and evaluated them on a Named Entity Recognition (NER) task. Despite the theoretical appeal of subword segmentation, word-level tokenization consistently outperformed all alternatives across all tokenization strategies tested. These findings suggest that in agglutinative, low-resource contexts, preserving boundaries via word-level tokenization may yield better embedding performance than complex statistical methods. This has practical implications for developing NLP pipelines for under-resourced languages where annotated data and computing power are limited.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。