开源17亿参数巴西葡萄牙语模型,可复现且评估严谨。
Manacá-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation
- 从头训练1.72亿参数模型,全流程容器化确保可复现。
- 在LAMBADA-PT上超越其他小模型,准确率提升显著。
- 揭示并修复分词器错误,适合关注语言模型公平评估的研究者。
巴西葡萄牙语在开源语言模型中仍被忽视,现有模型难以复现且缺乏不确定性度量。我们发布Manac'a-1B,一个1.72亿参数的解码器仅模型,专为巴西葡萄牙语训练,采用全容器化、可复现的流水线。预训练过程稳定,无跳过或NaN步骤,损失异常可自恢复,并公开完整日志与动态数据。在四个葡萄牙语基准上,与九个开源基线进行统一评估框架下的配对比较,每项结果均报告标准误与配对显著性检验,框架经已有文献验证。在最后一个词预测任务中,Manac'a-1B是7B以下最强模型,相较于Tucano-1b1和Tucano-2b4在LAMBADA-PT上表现更优;在常识补全任务中表现良好,在多项选择推理任务中接近随机水平,与其他小型基础模型一致。研究中还发现一个具体评估陷阱:将带大小写归一化的SentencePiece分词器转换为HuggingFace快速格式时,归一化器被静默丢失,导致所有大写词进入字节回退路径,显著降低得分,而该问题在聚合指标中不可见。未修正分词器使LAMBADA-PT准确率从45.3降至25.0,我们量化影响并提供一行修复方案,可精确复现训练时的分词器。代码、原始训练与评估日志、逐例预测向量、模型权重及修正后分词器均已公开,论文中所有数值均可重新计算。
原文摘要 · Abstract (English)
Brazilian Portuguese remains under-served by open language models, and the few that exist are difficult to reproduce and are often compared without measures of uncertainty. We release Manacá-1B, an open decoder-only model of 1.72 billion parameters trained from scratch for Brazilian Portuguese with a fully containerized, reproducible pipeline. The pretraining is stable, with zero skipped or NaN steps and self-recovering loss spikes, and we release its full log and dynamics. We evaluate the model against nine open baselines on four Portuguese benchmarks under a single harness. Every comparison reports a standard error and a paired significance test, and the harness is validated against previously published numbers. On last-word prediction Manacá-1B is the strongest model below the 7B scale, exceeding both Tucano-1b1 and Tucano-2b4 on LAMBADA-PT with large paired margins; it is competitive on commonsense completion and near chance on multiple-choice reasoning, as are all small base models. Along the way we document a concrete evaluation pitfall: converting a SentencePiece tokenizer with case-folding normalization to the HuggingFace fast format silently drops the normalizer, routing every capitalized token to byte-fallback and depressing scores in a way that is invisible in aggregate metrics. The uncorrected tokenizer lowered LAMBADA-PT accuracy from 45.3 to 25.0; we quantify the effect and provide a one-line fix that reproduces the training tokenizer exactly. Code, raw training and evaluation logs, per-example prediction vectors, the model weights, and the corrected tokenizer are released so that every number in this paper can be recomputed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。