从零训练微型罗马尼亚语模型,生成三百万条合成寓言故事。
TF3-RO-50M: Training Compact Romanian Language Models from Scratch on Synthetic Moral Microfiction
- 用语言学驱动的分词器和自建数据集,从头训练5165万参数模型。
- 压缩至2645万参数后仍保持良好表现,生成300万条罗马尼亚语寓言。
- 适合低资源语言研究者,可复现且支持本土化内容生成。
近期合成数据生成进展表明,当语料库结构可控且语言连贯时,小型语言模型可有效训练。然而,对于像罗马尼亚语这样形态复杂但计算资源匮乏的语言,尚无公开、端到端的完整流程来统一分词器设计、预处理、预训练、压缩、评估及大规模合成数据生成。基于英文寓言数据集TF1及其高质量罗马尼亚语翻译版本TF2,本文提出面向罗马尼亚语的模型训练框架TF3-RO,涵盖分词器训练、从零开始的模型开发与本土地域数据生成。该框架使用语言学指导的语料构建罗马尼亚专用BPE和Unigram分词器,缓解形态复杂导致的词元膨胀问题。通过长序列打包训练,我们从零预训练了一个51.65M参数的LLaMA风格Transformer。随后通过量化、结构化剪枝和基于对数概率的知识蒸馏进行优化,得到一个26.45M参数的紧凑学生模型,具备嵌入共享和良好部署特性。利用该蒸馏模型,TF3-RO通过受控组合提示框架生成三百万条本土地域罗马尼亚语寓言。整个流程整合了全面的评估体系,包含内在指标、罗马尼亚语一致性探测、实体连贯性、规则语法检查及大模型评估。TF3-RO为训练紧凑罗马尼亚语模型并生成大规模合成叙事语料提供了可复现且语言学基础坚实的框架。
原文摘要 · Abstract (English)
Recent advances in synthetic data generation have shown that compact language models can be trained effectively when the underlying corpus is structurally controlled and linguistically coherent. However, for morphologically rich and computationally under-resourced languages such as Romanian, there is still no openly documented, end-to-end pipeline that unifies tokenizer design, preprocessing, pretraining, compression, evaluation, and large-scale synthetic data generation in a reproducible framework. Building on TF1, a three-million-story English fable dataset, and TF2, which extends TF1 through high-quality Romanian translations, we introduce TF3-RO, a Romanian-centric language modeling pipeline spanning tokenizer training, from-scratch model development, and Romanian-native dataset generation. TF3-RO constructs Romanian-specific BPE and Unigram tokenizers from a linguistically informed corpus to mitigate token inflation induced by Romanian morphology. Using long-sequence packed training, we pretrain a 51.65M-parameter LLaMA-style Transformer entirely from scratch. The model is subsequently optimized through quantization, structured pruning, and logit-based knowledge distillation, yielding a compact 26.45M-parameter student model with tied embeddings and strong deployment characteristics. Using this distilled model, TF3-RO generates three million Romanian-native synthetic fables via a controlled combinatorial prompting framework. Across all stages, the pipeline integrates a comprehensive evaluation suite combining intrinsic metrics, Romanian agreement probes, entity coherence, rule-based grammar checking, and LLM-based assessment. TF3-RO provides a reproducible and linguistically grounded framework for training compact Romanian language models and producing large-scale synthetic narrative corpora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。