为黏着语设计新评估指标，发现现有方法常错切词干边界。
QuechuaTok: Morphological Boundary Accuracy as a Necessary Metric for Tokenizer Evaluation in Agglutinative Low-Resource Languages
- 提出形态边界准确率（MorphAcc）作为核心评估指标
- PRPE分词器形态准确率达83.33%，远超BPE的6.67%
- 适合研究低资源黏着语或分词器评估的学者参考
分词是自然语言处理的基础步骤，但标准评估指标如词频率无法反映黏着语的形态正确性。本文构建了QuechuaTok基准，对比四种分词策略——BPE、Unigram LM、WordPiece和一种形态感知的PRPE分词器——在南克丘亚语（quz）上的表现，该语言是南美洲800万至1000万人使用的低资源黏着语。基于20万句语料和SQUOIA有限状态形态分析器（Rios, 2016）作为银标准，评估三个指标：词频率、未知词率和形态边界准确率（MorphAcc）。结果表明，BPE在16k词表下词频率最低（1.636），但形态边界准确率仅6.67%；而PRPE达到83.33%的最高形态准确率，证明仅依赖词频率不足以评价黏着语分词器。所有代码与模型已公开于kaggle.com/code/macmaky/quechuatok。
原文摘要 · Abstract (English)
Tokenization is a foundational step in NLP pipelines, yet standard evaluation metrics such as fertility rate fail to capture morphological correctness for agglutinative languages. We present QuechuaTok, a systematic benchmark comparing four tokenization strategies - BPE, Unigram LM, WordPiece, and a morphology-aware PRPE tokenizer - for Southern Quechua (quz), a low-resource agglutinative language spoken by 8-10 million people in South America. Using a 200k-sentence corpus and the SQUOIA finite-state morphological analyzer (Rios, 2016) as silver standard, we evaluate three metrics: fertility rate, OOV rate, and morphological boundary accuracy (MorphAcc). Our results show that BPE achieves the lowest fertility rate (1.636 at 16k vocab) by memorizing surface word forms, while achieving only 6.67% MorphAcc. PRPE achieves 83.33% MorphAcc - the highest of all systems - demonstrating that fertility rate alone is insufficient to evaluate tokenizers for agglutinative languages. All code and models are publicly available at kaggle.com/code/macmaky/quechuatok
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。