轻量化适配罗马尼亚语的Llama2模型，资源少也能跑得快、效果好
RoQLlama: A Lightweight Romanian Adapted Language Model
- 用QLoRA量化技术训练，降低计算开销
- 在7个罗马尼亚下游任务中零样本表现持平或更优
- 适合资源有限但需罗马尼亚语NLP能力的研究者
近年来开源大语言模型在英语任务上取得了显著成果，但对罗马尼亚语等小语种支持不足。本文针对此问题，采用QLoRA技术训练轻量级模型，发布RoQLlama-7b。该模型在7个罗马尼亚下游任务的零样本设置下表现与全尺寸模型相当甚至更优，且在所有少样本提示中平均得分更高。同时，我们构建了首个罗马尼亚语医学问答数据集RoMedQA，包含单选题形式的医学问题。
原文摘要 · Abstract (English)
The remarkable achievements obtained by open-source large language models (LLMs) in recent years have predominantly been concentrated on tasks involving the English language. In this paper, we aim to advance the performance of Llama2 models on Romanian tasks. We tackle the problem of reduced computing resources by using QLoRA for training. We release RoQLlama-7b, a quantized LLM, which shows equal or improved results compared to its full-sized counterpart when tested on seven Romanian downstream tasks in the zero-shot setup. Also, it consistently achieves higher average scores across all few-shot prompts. Additionally, we introduce a novel Romanian dataset, namely RoMedQA, which contains single-choice medical questions in Romanian.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。