10亿参数模型让斯洛文尼亚语生成能力跃升，突破低资源语言瓶颈。
Generative Model for Less-Resourced Language with 1 billion parameters
- 基于英语OPT模型继续预训练，适配斯洛文尼亚语新分词器。
- 在句子简化任务上表现优于GPT-3.5-Turbo，分类任务略逊于专用BERT模型。
- 为低资源语言提供可扩展的生成式模型框架，适合多语言研究者参考。
大型语言模型（LLMs）是现代自然语言处理的基础设施。目前多数商业及开源模型（如ChatGPT、Llama、Falcon、Mistral）以英语文本训练为主，对低资源语言的知识和流畅性较弱。本文提出针对低资源语言的生成式语言模型——GaMS 1B（10亿参数），通过继续预训练现有英语OPT模型实现。我们开发了适配斯洛文尼亚语、克罗地亚语和英语的新分词器，并采用FOCUS与WECHSEL嵌入初始化方法迁移英语OPT模型的词向量。在斯洛文尼亚语基准测试套件多个分类任务及句子简化任务SENZA上进行评估，仅使用少样本上下文学习，未进行指令微调。分类任务中，生成模型表现不及已微调的斯洛文尼亚语BERT模型；但在句子简化任务上，其性能与GPT-3.5-Turbo相当或更优。
原文摘要 · Abstract (English)
Large language models (LLMs) are a basic infrastructure for modern natural language processing. Many commercial and open-source LLMs exist for English, e.g., ChatGPT, Llama, Falcon, and Mistral. As these models are trained on mostly English texts, their fluency and knowledge of low-resource languages and societies are superficial. We present the development of large generative language models for a less-resourced language. GaMS 1B - Generative Model for Slovene with 1 billion parameters was created by continuing pretraining of the existing English OPT model. We developed a new tokenizer adapted to Slovene, Croatian, and English languages and used embedding initialization methods FOCUS and WECHSEL to transfer the embeddings from the English OPT model. We evaluate our models on several classification datasets from the Slovene suite of benchmarks and generative sentence simplification task SENTA. We only used a few-shot in-context learning of our models, which are not yet instruction-tuned. For classification tasks, in this mode, the generative models lag behind the existing Slovene BERT-type models fine-tuned for specific tasks. On a sentence simplification task, the GaMS models achieve comparable or better performance than the GPT-3.5-Turbo model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。