精选教育类语料能让小模型超越大模型，证明数据质量比数量更重要。
Curió-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining
- 用100亿葡萄牙语令牌训练70亿参数模型，再筛选出10亿教育/科技文本微调
- 小样本高质量数据训练的模型在评测中超过全量数据训练版本
- 适合关注低资源语言适配、教育AI的开发者和研究者
持续预训练通过向模型注入更多特定语言或领域的数据来扩展其能力，是适应通用模型至新场景的高效替代方案。本文基于LLaMA-2构建70亿参数的Curió 7B模型，使用来自ClassiCC-PT语料库的1000亿葡萄牙语令牌进行训练——这是目前超过30亿参数规模下最大规模的葡萄牙语专用持续预训练工作。除规模外，我们探究了数量是否足够，还是数据质量在语言适配中起决定性作用。为此，我们提出Curió-Edu 7B，仅使用同一语料库中教育与STEM过滤子集的100亿令牌进行训练。尽管仅使用10%的数据量和20%的计算资源，该模型在评估中仍优于全量数据训练模型，表明即使在目标语言先验知识有限的情况下，数据选择也至关重要。相关模型已在Hugging Face发布。
原文摘要 · Abstract (English)
Continued pretraining extends a language model's capabilities by further exposing it to additional data, often tailored to a specific linguistic or domain context. This strategy has emerged as an efficient alternative to full retraining when adapting general-purpose models to new settings. In this work, we investigate this paradigm through Curió 7B, a 7-billion-parameter model derived from LLaMA-2 and trained on 100 billion Portuguese tokens from the ClassiCC-PT corpus - the most extensive Portuguese-specific continued-pretraining effort above the three-billion-parameter scale to date. Beyond scale, we investigate whether quantity alone suffices or whether data quality plays a decisive role in linguistic adaptation. To this end, we introduce Curió-Edu 7B, a variant trained exclusively on the educational and STEM-filtered subset of the same corpus, totaling just 10 billion tokens. Despite using only 10% of the data and 20% of the computation, Curió-Edu 7B surpasses the full-corpus model in our evaluations, demonstrating that data selection can be fundamental even when adapting models with limited prior exposure to the target language. The developed models are available at https://huggingface.co/collections/ClassiCC-Corpus/curio-edu
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。