3.8B小模型通过课程学习实现波斯语高效适配
Persian-Phi: Efficient Cross-Lingual Adaptation of Compact LLMs via Curriculum Learning
- 用双语故事预训练对齐嵌入，再用参数高效微调
- 在波斯语榜单上表现媲美更大模型，仅需少量资源
- 适合想低成本拓展低资源语言的开发者使用
当前大型语言模型（LLM）在低资源语言上的训练成本高昂，阻碍了AI普惠。本文提出3.8B参数的Persian-Phi模型，挑战了多语言能力需大模型或多语言基线的假设。我们通过一种资源高效的课程学习流程，将原本单语英文的Microsoft Phi-3 Mini成功适配至波斯语。方法包括：先用双语叙事数据集Tiny Stories进行嵌入对齐的“热身”阶段，随后进行持续预训练和指令微调，采用参数高效微调（PEFT）。尽管模型小巧，Persian-Phi在HuggingFace的Open Persian LLM Leaderboard上表现优异。研究验证了一种可扩展、低资源的先进LLM跨语言适配框架。模型已公开于https://huggingface.co/amirakhlaghiqqq/PersianPhi。
原文摘要 · Abstract (English)
The democratization of AI is currently hindered by the immense computational costs required to train Large Language Models (LLMs) for low-resource languages. This paper presents Persian-Phi, a 3.8B parameter model that challenges the assumption that robust multilingual capabilities require massive model sizes or multilingual baselines. We demonstrate how Microsoft Phi-3 Mini -- originally a monolingual English model -- can be effectively adapted to Persian through a novel, resource-efficient curriculum learning pipeline. Our approach employs a unique "warm-up" stage using bilingual narratives (Tiny Stories) to align embeddings prior to heavy training, followed by continual pretraining and instruction tuning via Parameter-Efficient Fine-Tuning (PEFT). Despite its compact size, Persian-Phi achieves competitive results on Open Persian LLM Leaderboard in HuggingFace. Our findings provide a validated, scalable framework for extending the reach of state-of-the-art LLMs to underrepresented languages with minimal hardware resources. The Persian-Phi model is publicly available at https://huggingface.co/amirakhlaghiqqq/PersianPhi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。