arXiv:2601.11643cs.CYcs.AI2026-01被引 3

为印尼语设计基于音节的分词方法,提升语言模型效率与准确性。

Syllabic Agglutinative Tokenizations for Indonesian LLM: A Study from Gasing Literacy Learning System

  • 按音节切分文本后结合字节对编码,构建符合语言结构的3500词表。
  • 在印尼维基和民俗语料上,瑞尼效率达0.74,高于多语言模型的0.50-0.64。
  • 适合研究形态丰富、资源少的语言,尤其关注印尼语教育技术的应用。

本文提出一种受印尼语识字教学系统启发的音节基础分词方法,基于信息论原则,在字节对编码前按音节边界分割印尼语文本,构建与语言音形结构一致的词汇表。通过规则化音节识别高频单元,生成仅3500个词条的紧凑词库,同时保留字符级回退机制以保证覆盖度。在印尼维基百科及印尼文化数字图书馆(PDBI)民俗语料上的实证评估显示,该方法相比传统分词方式显著提升性能:瑞尼效率达0.74,优于预训练多语言模型的0.50–0.64;尽管词表规模小一个数量级,平均词长仍达3.67字符,高于GPT-2的2.72字符。这一优势源于将字符级依赖关系内嵌于音节单位中,减轻语言模型计算负担,同时尊重印尼语的黏着性形态特征。由此构建的模型称为TOBA LLM(基于黏着性的最优分词),体现了人类识字教学法与计算优化的融合,为处理形态复杂且资源稀缺的自然语言提供了新范式。

原文摘要 · Abstract (English)

This paper presents a novel syllable-based tokenization approach for Indonesian large language models, inspired by the Gasing Literacy Learning System's pedagogical methodology. Drawing on information-theoretic principles, we develop a tokenization framework that segments Indonesian text at syllable boundaries before applying byte-pair encoding, creating a vocabulary that aligns with the language's morphophonological structure. Our approach first identifies high-frequency syllables through rule-based segmentation, then constructs a compact vocabulary of 3,500 tokens that preserves meaningful linguistic units while maintaining coverage through character-level fallback. Empirical evaluation on Indonesian Wikipedia and folklore corpora from Indonesian Culture Digital Library (PDBI) demonstrates substantial improvements over conventional tokenization methods: the syllable-based approach achieves Rényi efficiency of 0.74 compared to 0.50-0.64 for pretrained multilingual tokenizers, while maintaining higher average token lengths (3.67 characters versus 2.72 for GPT-2) despite using a vocabulary an order of magnitude smaller. These gains emerge from the method's ability to internalize character-level dependencies within syllable units, reducing the computational burden on language models while respecting Indonesian's agglutinative morphology. We call the LLM built upon this principle, TOBA LLM (Tokenisasi Optimum Berbasis Aglutinasi), the convergence of human literacy pedagogy with computational optimization principles offers a promising paradigm for developing linguistically-informed tokenization strategies, particularly for morphologically rich and underrepresented languages in natural language processing.

分词印尼语形态学语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。