用单块显卡训练30亿参数的萨丁尼亚语模型，解决小语种语言模型缺失问题。
LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language
- 基于Qwen2.5-3B在24GB显卡上持续预训练与微调，适配萨丁尼亚语
- rsLoRA r256使英译萨丁尼亚语达28.5 BLEU，显著优于全量微调
- 发现适配器容量比具体方法更重要，且翻译指标无法捕捉所有错误模式
萨丁尼亚语是约一百万使用者的罗曼语，现代自然语言处理中几乎无存在感。商业服务不支持该语言，现有语言模型也无法可靠生成。本文提出LLiMba，一个30亿参数的萨丁尼亚语适配模型，基于Qwen2.5-3B-Instruct，通过在单张24GB消费级显卡上进行持续预训练（CPT）和监督微调（SFT）。语料库包含1150万词元的萨丁尼亚语（涵盖LSC、Logudorese和Campidanese），并辅以240万词元相关罗曼语文本作为回放，以缓解语体模糊。经CPT后，模型在保留萨丁尼亚语上的困惑度为6.76，且在六项FLORES-200方向上均优于基线。五种SFT配置比较显示：rsLoRA r256在所有萨丁尼亚语方向上表现最佳，英译萨丁尼亚语达28.5 BLEU，高于CPT后的17.3和全量微调的21.0。秩消融实验表明，r128介于LoRA r64与rsLoRA r256之间，但暴露其他变体未出现的跨脚本泄漏问题。低秩方法（如LoRA r64）在微调中保留更少事实内容，产生更多自信的虚构内容，尽管所有方法均会生成训练数据外的内容。DoRA r256虽训练评估差距最小，但事实准确性最差。结果表明，适配器容量比方法选择更重要，更强正则化并非始终有益，且翻译指标无法反映不同配置间质的差异。跨脚本困惑度比较需考虑字节回退分词对非拉丁脚本的压制效应。
原文摘要 · Abstract (English)
Sardinian, a Romance language with roughly one million speakers, has minimal presence in modern NLP. Commercial services do not support it, and current language models do not produce it reliably. We present LLiMba, a 3B parameter Sardinian-ready model adapted from Qwen2.5-3B-Instruct through continued pretraining (CPT) and supervised fine-tuning (SFT) on a single 24 GB consumer GPU. The corpus contains 11.5 million tokens of Sardinian spanning LSC, Logudorese, and Campidanese, augmented with 2.4 million tokens of related Romance text as replay against register blurring. After CPT the model reaches a perplexity of 6.76 on held out Sardinian and outperforms the base across all six FLORES-200 directions. We compare five SFT configurations under matched conditions: full fine-tuning, LoRA r64, rsLoRA r128, rsLoRA r256, and DoRA r256. rsLoRA r256 wins on every direction into Sardinian, reaching 28.5 BLEU from English against 17.3 after CPT and 21.0 with full fine-tuning. The rank ablation places r128 between LoRA r64 and rsLoRA r256 on BLEU but reveals failure modes invisible to the metric, including leakage across scripts no other variant produces. LoRA r64 retains less factual content from SFT than configurations at higher rank and produces more confident fabrications, though all methods fabricate on content absent from training. DoRA r256 yields the smallest gap between training and evaluation but the worst factual accuracy. The findings indicate that adapter capacity matters more than the choice among LoRA variants for adapting a Romance pretrained base to a low resource Romance target, that stronger regularization is not uniformly beneficial, and that translation metrics smoothly order configurations whose qualitative behavior differs categorically. Perplexity comparisons across scripts must account for byte fallback tokenization, which deflates the metric for scripts other than Latin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。