用词法分段方法做分词，效果不输主流方案
Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenization
- 用Morfessor和Seq2Seq模型进行词法分段，作为分词器组件
- 词法分段可达到与主流分词器相当的性能
- 词频分布均衡的词表更利于语言模型表现
本文介绍了团队Ryu对原定于SIGMORPHON 2024的子词分词共享任务的提交。研究探索了词法分段方法能否作为子词分词器的一部分。采用两种方法：基于统计的分段工具Morfessor和基于Transformer的序列到序列（seq2seq）分段模型。实验结果表明，词法分段在性能上可媲美常用的子词分词器。此外，研究还分析了分词器词表对语言模型性能的影响，发现词频分布均衡的词表表现更优。通过将高频词保持为独立词元，可实现平衡的词表结构。
原文摘要 · Abstract (English)
This papers presents the submission of team Ryu to the canceled SIGMORPHON 2024 shared task on subword tokenization. My submission explores whether morphological segmentation methods can be used as a part of subword tokenizers. I adopt two approaches: the statistical segmentation method Morfessor and a transformer based sequence-to-sequence (seq2seq) segmentation model in tokenizers. The prediction results show that morphological segmentation could be as effective as commonly used subword tokenizers. Additionally, I investigate how a tokenizer's vocabulary influences the performance of language models. A tokenizer with a balanced token frequency distribution tends to work better. A balanced token vocabulary can be achieved by keeping frequent words as unique tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。