518M参数阿拉伯语专用模型，小体积高效率，性能超同类开源模型。
RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment
- 通过注入2.7万阿拉伯语词元并微调，构建专用小模型。
- 在3个基准测试中平均准确率达35.9%，接近9B模型的67%性能。
- 可部署于单张H100，推理速度达635词元/秒，适合边缘设备。
当前开源阿拉伯语大模型分为两类：一类是<10亿参数的多语言模型，将阿拉伯语作为次要语言（如Qwen2.5-0.5B、Falcon-H1-0.5B）；另一类是70亿至700亿参数的专用模型，需服务器运行（如Jais、AceGPT、ALLaM、SILMA）。此前唯一公开的<20亿参数阿拉伯语专用模型Kuwain-1.5B未发布权重。我们提出RightNow-Arabic-0.5B-Turbo，一个基于Qwen2.5-0.5B的518M参数解码器语言模型。通过均值子词初始化注入27,032个阿拉伯语词元，在8张H100上使用FSDP、FlashAttention变长打包和Liger融合内核继续预训练5.04亿阿拉伯语词元，并在129,116组阿拉伯语指令对上进行监督微调（响应仅损失掩码），在6,750组偏好对上应用直接偏好优化，最后对三个检查点进行权重汤融合。在lm-evaluation-harness的三个阿拉伯语基准测试（COPA-ar、Arabic HellaSwag、ArabicMMLU）中，融合模型达到35.9%平均准确率，优于所有同规模开源模型，与Falcon-H1-1.5B在COPA-ar上持平（58.4%），但仅为其1/3大小，并恢复了SILMA-9B平均性能的67%，仅需其1/18参数量。边缘版本量化至398 MB（q4_k_m），在单张H100上以批量大小1实现635词元/秒的推理速度。所有代码（25个脚本共5,555行）、权重（bf16、int8及四种GGUF量化格式）和评测脚本已公开于https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo。
原文摘要 · Abstract (English)
Open Arabic large language models split into two classes: sub-1B multilingual models that treat Arabic as an afterthought (Qwen2.5-0.5B, Falcon-H1-0.5B), and 7B-70B Arabic-specialized models that require a server to run (Jais, AceGPT, ALLaM, SILMA). The one published attempt at a sub-2B Arabic-specialized model, Kuwain-1.5B, never released its weights. We present RightNow-Arabic-0.5B-Turbo, a 518M-parameter Arabic-specialized decoder LLM built on Qwen2.5-0.5B. The pipeline adds 27,032 Arabic tokens via mean-subtoken initialization, continues pretraining on 504M Arabic tokens on 8xH100 with FSDP, FlashAttention varlen packing, and Liger fused kernels, then applies supervised fine-tuning on 129,116 Arabic instruction pairs with response-only loss masking, direct preference optimization on 6,750 Arabic preference pairs, and weight soup merging across three checkpoints. On three lm-evaluation-harness Arabic benchmarks (COPA-ar, Arabic HellaSwag, ArabicMMLU) the merged model reaches 35.9% mean accuracy, beats every same-class open model, ties Falcon-H1-1.5B on COPA-ar (58.4%) at one-third the size, and recovers 67% of SILMA-9B's mean at 1/18 the parameters. The edge build quantizes to 398 MB (q4_k_m) and delivers 635 tokens/s at batch size 1 on a single H100 via llama.cpp. All code (5,555 lines across 25 scripts), weights (bf16, int8, and four GGUF quantizations), and benchmark scripts are released at https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。