对比三款大模型在尼泊尔语罗马字上的适配效果，发现微调后性能显著提升。
Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali
- 用1万条双语指令数据微调三个80亿参数模型，仅训练1%参数。
- 微调后所有模型的BERTScore达0.75，chrF++超23，实现流畅生成。
- 适合低资源语言研究者参考，尤其关注模型迭代优化的场景。
罗马化尼泊尔语（用拉丁字母书写的尼泊尔语）是尼泊尔非正式数字交流的主要形式，但大型语言模型领域仍严重缺乏资源。本研究系统评估了三个规模相近的开源模型——Llama-3.1-8B、Mistral-7B-v0.1和Qwen3-8B——在罗马化尼泊尔语上的语言适应能力。基于包含10,000条音译指令样本的定制双语数据集，在零样本与微调两种设置下进行评估。性能通过五项指标覆盖七维测量：困惑度（PPL）、BERTScore、chrF++、ROUGE-1、ROUGE-2、ROUGE-L 和 BLEU，涵盖流利度、语音一致性与语义完整性。使用量化低秩适应（QLoRA）结合秩稳定LoRA（rsLoRA，rank=32），在双张NVIDIA Tesla T4 GPU上仅训练约1%参数，总耗时不足27小时。零样本阶段，三模型均无法生成罗马化尼泊尔语，表现出各具特色的架构缺陷。微调后，三模型均克服上述问题，达到平均BERTScore 0.75，chrF++ > 23。整体十项标准评估显示，Qwen3-8B为最优架构，是唯一在零样本阶段产生语义相关输出并微调后在结构对齐指标上领先的模型。适应潜力假设得到验证：尽管Llama-3.1-8B零样本表现最差，其在困惑度（Δ = -49.77）和BERTScore（Δ = +0.3287）上的绝对提升最大，适合作为低资源迭代开发流程首选。
原文摘要 · Abstract (English)
Romanized Nepali, the Nepali language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal, yet it remains critically underresourced in the landscape of Large Language Models (LLMs). This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models: Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B. We evaluate these architectures under zero-shot and fine-tuned settings using a curated bilingual dataset of 10,000 transliterated instruction-following samples. Performance is quantified across five metrics spanning seven measurement dimensions: Perplexity (PPL), BERTScore, chrF++, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU, capturing fluency, phonetic consistency, and semantic integrity. Models were fine-tuned using Quantized Low-Rank Adaptation (QLoRA) with Rank-Stabilized LoRA (rsLoRA) at rank r=32 on dual NVIDIA Tesla T4 GPUs, training only approximately 1% of each model's parameters in under 27 total GPU-hours. At zero-shot, all three models fail to generate Romanized Nepali, each exhibiting a distinct architecture-specific failure mode. Following fine-tuning, all three resolve these failures and converge to BERTScore approximately 0.75 and chrF++ greater than 23. Overall dimension-wise assessment across ten criteria identifies Qwen3-8B as the overall recommended architecture, being the only model to produce semantically relevant zero-shot output and leading all structural alignment metrics post-SFT. The adaptation headroom hypothesis is confirmed: Llama-3.1-8B, despite its weakest zero-shot baseline, achieves the largest absolute fine-tuning gains in PPL (Delta = -49.77) and BERTScore (Delta = +0.3287), making it the preferred choice for iterative low-resource development pipelines. This work establishes the first rigorous baseline for Romanized Nepali adaptation in comparable-sized open-weight LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。