arXiv:2608.30092cs.CLcs.AI2026-08

开源10亿参数英尼双语模型,专为天城文优化分词器。

Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer

  • 从头训练,使用自研天城文感知分词器和纯代码训练栈。
  • 在英语科学问答任务上表现超同类模型,尼泊尔语理解能力被格式误导低估。
  • 发布基础与指令微调模型,适合低资源语言研究者使用。

我们提出Arkios,一个10.4亿参数的稠密Transformer模型,从零开始在1500亿个标记的英尼双语文本上预训练,采用自研单文件C/CUDA训练栈及专为本项目设计的天城文感知字节级BPE分词器。在ARC-Easy和ARC-Challenge任务中,尽管训练样本仅为同类开源模型的十分之一,其表现仍优于Pythia-1.4B、TinyLlama-1.1B和OLMo-1B。这一优势可能源于预训练数据(教育类网页文本)与测试任务(小学科学题)的高度匹配,而非通用能力。我们报告了标准评估结果,并修正了早期部分样本估算;发现小模型在低资源语言上的评测存在陷阱:标准字母选项提示格式使模型在尼泊尔语和英语阅读理解中均表现随机(尼泊尔语0.240,英语0.236,随机基线为0.250),但直接评分答案文本揭示真实理解力(尼泊尔语0.306,英语0.387)。我们引入一种情境化工具使用协议,在指令微调中仅当上下文声明工具清单时才允许调用工具。所有模型权重按Apache-2.0开源,训练代码与部分私有尼泊尔语语料未公开,但复现报告结果所需内容均已提供。

原文摘要 · Abstract (English)

We present Arkios, a 1.04B-parameter dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text, using a custom single-file C/CUDA training stack and a Devanagari-aware byte-level BPE tokenizer built for this project. On ARC-Easy and ARC-Challenge, Arkios exceeds three comparably sized open models (Pythia-1.4B, TinyLlama-1.1B, OLMo-1B) despite an order of magnitude fewer training tokens, likely aided by a match between our educational-web-text pretraining data and ARC's grade-school-science format rather than a general capability advantage. We report full evaluation results under standard protocols, including a correction to an earlier partial-sample estimate, and findings specific to evaluating small models in a low-resource language: the standard multiple-choice-letter prompt format used by common evaluation harnesses places this model at chance on Nepali reading comprehension, and simultaneously at chance on English in the same format, which would lead a naive benchmark run to conclude the model has no Nepali ability when in fact it does. Concretely, both languages score at chance in the letter-choice format (0.240 Nepali, 0.236 English, against a chance baseline of 0.250), while scoring the answer text directly reveals genuine, English-favoring comprehension (0.306 Nepali, 0.387 English). We describe a manifest-conditioned tool-use contract introduced during instruction tuning, where tool calls are permitted only when a tool manifest is declared in context and suppressed otherwise, and report where that contract holds and where it does not. We release both the base and instruction-tuned model weights under Apache-2.0. The training code and a small privately-sourced portion of the Nepali pretraining corpus are not released; everything needed to reproduce the reported numbers from the released weights is included here.

双语模型低资源语言天城文开源模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。