为尼泊尔语构建高效GPT模型，用自研分词器提升生成质量。
Towards Nepali-language LLMs: Efficient GPT training with a Nepali BPE tokenizer
- 基于GPT-2架构，使用自研16k BPE分词器处理尼泊尔语文本。
- 在10.75GB语料上训练两轮，最终困惑度达21.80，可生成连贯新闻文本。
- 适合关注低资源语言、尼泊尔语技术落地的研究者与开发者。
尼泊尔语是全球超过3200万人使用的低资源语言，因语法复杂、黏着性构词和高质量语料稀缺，在自然语言处理中仍面临挑战。现有研究多集中于基础编码器架构，难以满足尼泊尔语的文本生成需求。本文提出一种基于GPT-2的尼泊尔语语言模型，采用受GPT-3启发的训练策略，包括优化学习率调度、批量规模扩展及结构改进。专为尼泊尔语文本训练了16,000个token的字节对编码（BPE）分词器，以实现更一致的分词效果和更好的输入表示。模型在包含10.75GB清洗后的NepBERTa语料和额外网络抓取尼泊尔新闻文章的合并数据集上预训练。引入FlashAttention以降低内存占用并稳定训练过程。经过两轮训练，模型达到训练损失3.168177，验证损失3.081982，最终困惑度21.80，表明其具备生成连贯尼泊尔语新闻风格文本的能力。
原文摘要 · Abstract (English)
Nepali, a low-resource language spoken by over 32 million people, continues to face challenges in natural language processing (NLP) due to its complex grammar, agglutinative morphology, and limited availability of high-quality corpora. Most efforts to date have centered on basic encoder architectures; they remain insufficient for Nepali-specific text generation. This study presents a GPT-2-based Nepali language model trained using several training strategies inspired by GPT-3, including optimized learning rate schedules, batch scaling, and architectural refinements. A custom 16k Byte-Pair Encoding (BPE) tokenizer was trained exclusively on Nepali text to ensure more consistent segmentation and improved input representation. The model was pretrained on a combined dataset comprising a 10.75GB cleaned NepBERTa corpus and additional web-scraped Nepali news articles. FlashAttention was integrated to reduce memory usage and stabilize training. After two epochs, the model achieved a training loss of 3.168177, a validation loss of 3.081982, and a final perplexity of 21.80, demonstrating its capability to generate coherent Nepali news-style text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。