系统评估土耳其语子词分词策略,揭示数据、词汇量与形态结构的协同关系。
Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay
- 控制数据与词汇量变量,对比词级、形态级和字符级分词器性能。
- 在多个任务上发现形态级分词在语法和形态敏感任务中表现最优。
- 提供可复现的诊断工具包,适合研究形态丰富语言的NLP开发者。
子词分词是形态丰富语言(如土耳其语)神经语言建模中的关键设计选择,其高度黏着性对词汇效率和形态保真度构成挑战。以往研究通常未系统控制训练语料规模,缺乏内在诊断能力,且评估任务范围有限。本文首次开展系统性、原则性的土耳其语子词分词研究,构建‘子词手册’:联合调节词汇量与训练语料规模(数据-词汇耦合),在统一参数预算下比较WordPiece、形态层级与字符基线分词器,并在语义(自然语言推断、语义相似度、情感分析、命名实体识别)、句法(词性标注、依存句法分析)及形态敏感探测任务上评估。为解释分词器成败原因,提出形态感知诊断工具包,包含边界级微观/宏观F1、词干原子性与表面边界命中率分离、过/欠分割指数、字符/词编辑距离(CER/WER)、延续率、词缀类型覆盖度与词级原子性等指标。贡献包括:(i) 数据-词汇-效果三元关系的系统探究;(ii) 融合内在诊断与外在任务的统一评估框架;(iii) 明确字符与形态级分词在特定场景下的优势;(iv) 开源评估代码、分词管道与模型。本工作为形态丰富语言的分词器构建提供可操作指导,并建立可复现的研究基础。
原文摘要 · Abstract (English)
Tokenization is a pivotal design choice for neural language modeling in morphologically rich languages (MRLs) such as Turkish, where productive agglutination challenges both vocabulary efficiency and morphological fidelity. Prior studies have explored tokenizer families and vocabulary sizes but typically (i) vary vocabulary without systematically controlling the tokenizer's training corpus, (ii) provide limited intrinsic diagnostics, and (iii) evaluate a narrow slice of downstream tasks. We present the first comprehensive, principled study of Turkish subword tokenization; a "subwords manifest", that jointly varies vocabulary size and tokenizer training corpus size (data and vocabulary coupling), compares multiple tokenizer families under matched parameter budgets (WordPiece, morphology level, and character baselines), and evaluates across semantic (NLI, STS, sentiment analysis, NER), syntactic (POS, dependency parsing), and morphology-sensitive probes. To explain why tokenizers succeed or fail, we introduce a morphology-aware diagnostic toolkit that goes beyond coarse aggregates to boundary-level micro/macro F1, decoupled lemma atomicity vs. surface boundary hits, over/under-segmentation indices, character/word edit distances (CER/WER), continuation rates, and affix-type coverage and token-level atomicity. Our contributions are fourfold: (i) a systematic investigation of the vocabulary-corpus-success triad; (ii) a unified, morphology-aware evaluation framework linking intrinsic diagnostics to extrinsic outcomes; (iii) controlled comparisons identifying when character-level and morphology-level tokenization pay off; and (iv) an open-source release of evaluation code, tokenizer pipelines, and models. As the first work of its kind, this "subwords manifest" delivers actionable guidance for building effective tokenizers in MRLs and establishes a reproducible foundation for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。