改进阿拉伯语等非连写语言的分词，提升模型对词干构词的识别能力。
Splintering Nonconcatenative Languages for Better Tokenization
- 将非连写语言文本重排为更符合词根模板结构的线性形式
- 在希伯来语、阿拉伯语和马来语中降低子词词汇量30%以上
- 适用于需要精准处理形态变化的语言，如学术研究或NLP工具开发
常见的子词分词算法（如BPE和UnigramLM）假设文本可通过拼接方式分割成有意义单元，但对希伯来语、阿拉伯语这类通过词根-模板模式编码形态的语言，或马来语、格鲁吉亚语中常见前缀后缀分离的情况并不适用。本文提出SPLINTER，一种预处理步骤，将文本重排为更贴近非连写形态的线性形式，使分词器能发现更有意义的连续片段。我们在希伯来语、阿拉伯语和马来语上通过内在指标评估了词表表现，并在基于BERT架构的希伯来语模型上验证了其下游任务有效性。
原文摘要 · Abstract (English)
Common subword tokenization algorithms like BPE and UnigramLM assume that text can be split into meaningful units by concatenative measures alone. This is not true for languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. We present SPLINTER, a pre-processing step which rearranges text into a linear form that better represents such nonconcatenative morphologies, enabling meaningful contiguous segments to be found by the tokenizer. We demonstrate SPLINTER's merit using both intrinsic measures evaluating token vocabularies in Hebrew, Arabic, and Malay; as well as on downstream tasks using BERT-architecture models trained for Hebrew.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。