针对黏着语设计语法优先分词，提升语言模型处理效率
VerChol -- Grammar-First Tokenization for Agglutinative Languages
- 基于语法结构而非字节对进行分词，保留词素边界
- 在泰米尔语等黏着语上减少40%以上分词数量
- 适合多词素语言的模型训练与部署
分词是所有大语言模型流程的基础步骤，但主流的字节对编码（BPE）及其变体本质上不依赖书写系统，且针对英语类形态优化。对于黏着语这一语言类型（涵盖德拉维达语系、突厥语族、乌拉尔语系、韩语、日语、斯瓦希里语、巴斯克语等），一个词可能将词根、时态、体、人称、数、性、格、后置词等信息整合为一个书写单元。统计分词方法会将这些词切分为字节对片段，破坏词素边界并显著增加分词数量。
原文摘要 · Abstract (English)
Tokenization is the foundational step in all large language model (LLM) pipelines, yet the dominant approach Byte Pair Encoding (BPE) and its variants is inherently script agnostic and optimized for English like morphology. For agglutinative languages a typological class encompassing the Dravidian family (Tamil, Kannada, Telugu, Malayalam), Turkic languages (Turkish, Azerbaijani, Uzbek), Uralic languages (Finnish, Hungarian, Estonian), Korean, Japanese, Swahili, Basque, and others, a single word may encode root, tense, aspect, person, number, gender agreement, case, and postpositions into one orthographic unit. Statistical tokenizers fragment these words into byte pair chunks that sever morpheme boundaries and inflate token counts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。