arXiv:2512.18399cs.CLcs.AI2025-12

为阿拉伯语优化的分词器,提升模型压缩效率与训练速度。

AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3

  • 用归一化流水线处理阿拉伯语拼写变体,结合SentencePiece算法
  • 分词后每词平均1.199个标记,比基线低18%的词频膨胀
  • 可快速集成到Qwen3模型,适合做阿拉伯语NLP研究者

分词是大语言模型的关键预处理步骤,直接影响训练效率与下游表现。通用分词器主要基于英语和拉丁文字母训练,在形态丰富的语言如阿拉伯语上表现不佳,导致标记序列过长、压缩效率下降。本文提出AraToken,一种基于SentencePiece Unigram算法的阿拉伯语优化分词器,配备完整的归一化流水线,解决阿尔法变体、元音符号和阿拉伯数字等书写差异。我们系统比较了BPE、WordPiece与SentencePiece在多种配置下的表现,发现带归一化的SentencePiece将词频膨胀降低18%(1.199对1.35标记/词)。此外,提出语言扩展流水线(LEP),通过词汇扩展与均值子词初始化,结合选择性解冻Transformer层,将Qwen3-0.6B在10万条阿拉伯语样本上的评估损失从8.28降至2.43,仅需800步训练。我们已公开分词器、训练脚本与模型权重,以推动阿拉伯语自然语言处理研究。

原文摘要 · Abstract (English)

Tokenization is a critical preprocessing step for large language models (LLMs), directly impacting training efficiency and downstream performance. General-purpose tokenizers trained predominantly on English and Latin-script languages exhibit suboptimal performance on morphologically rich languages such as Arabic, resulting in inflated token sequences and reduced compression efficiency. In this work, we present AraToken, an Arabic-optimized tokenizer built on SentencePiece Unigram algorithm with a comprehensive normalization pipeline addressing Arabic-specific orthographic variations including Alif variants, diacritics, and Arabic-Indic numerals. We systematically compare BPE, WordPiece, and SentencePiece algorithms across multiple configurations, demonstrating that SentencePiece with normalization achieves 18% lower fertility (1.199 vs 1.35 tokens/word) compared to unnormalized baselines. Furthermore, we introduce the Language Extension Pipeline (LEP), a method for integrating the optimized tokenizer into Qwen3-0.6B through vocabulary extension with mean subtoken initialization and selective transformer layer unfreezing. Our experiments show that LEP reduces evaluation loss from 8.28 to 2.43 within 800 training steps on 100K Arabic samples. We release our tokenizer, training scripts, and model checkpoints to facilitate Arabic NLP research.

分词优化阿拉伯语Qwen3语言扩展

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。