首个专为印地语设计的大语言模型,填补低资源语言空白。
HindiLLM: Large Language Model for Hindi
- 分两步训练:先构建高质量语料库,再用字节对编码进行预训练
- 在情感分析等任务上超越现有模型,性能提升显著
- 适合研究印地语NLP或低资源语言应用的开发者
大型语言模型的发展推动了自然语言处理的进步,但多数研究集中于英语,导致印地语等印度语言缺乏高性能模型。本文提出两个自回归大语言模型:HindiLLM-Small和HindiLLM-Medium。采用两阶段流程:首先构建大规模高质量文本语料库用于无监督预训练;其次基于该数据训练专用的字节对编码分词器(HindiLLM tokenizer),并在未标注数据上完成预训练,获得基础模型。随后在多个主流标注数据集上对基础模型进行有监督微调,涵盖情感分析、文本分类、自然语言推理及多选问答任务,评估其实际表现。实验表明,微调后的HindiLLM模型在多数任务中优于现有模型。
原文摘要 · Abstract (English)
The advancements in the Large Language Model (LLM) have helped in solving several problems related to language processing. Most of the researches have focused on the English language only, because of its popularity and abundance on the internet. However, a high-performance language model for Hindi and other Indic languages is lacking in the literature. In this work, we have pre-trained two autoregressive LLM models for the Hindi language, namely HindiLLM-Small and HindiLLM-Medium. We use a two-step process comprising unsupervised pre-training and supervised fine-tuning. First, we create a large and high-quality text corpus for unsupervised pre-training. Next, we train a Byte-Pair Encoding, named HindiLLM tokenizer, using the pre-training text data. We then perform training on the unlabeled data, known as the pre-training step, to get the HindiLLM base models. Furthermore, we perform fine-tuning of the HindiLLM base models for different tasks like sentiment analysis, text classification, natural language inference, and multiple choice question-answer on popular labeled datasets to measure the real-world performance. The evaluation shows that the HindiLLM-based fine-tuned models outperform several models in most of the language related tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。