开源30亿参数模型,系统探索预训练数据处理对模型能力的影响。
daVinci-LLM:Towards the Science of Pretraining
- 采用数据达尔文主义框架,分阶段过滤与合成数据,提升处理深度。
- 在8万亿词上训练30亿参数模型,证明数据处理深度比单纯扩容更关键。
- 适合研究预训练机制、模型可扩展性及开放科学的学者与工程师。
基础预训练阶段决定了模型的能力上限,而微调难以弥补预训练奠定的基础,但该阶段仍严重缺乏系统研究。这源于结构性矛盾:拥有算力资源的机构受商业压力影响,难于公开透明;学术机构虽有研究自由,却缺乏预训练规模的算力。daVinci-LLM占据这一未被探索的交叉地带,结合工业级算力与完全的研究自由,推动预训练科学的发展。我们采用全开源范式,将开放视为科研方法论,发布完整数据处理流程、全部训练过程与系统性探索结果。针对领域内缺乏系统化数据处理方法的问题,提出数据达尔文主义框架(Data Darwinism),涵盖从筛选到合成的L0-L9层级分类。我们从随机初始化训练一个30亿参数模型,使用两阶段自适应课程,在8万亿词上逐步由基础能力过渡到推理增强。通过200余次受控消融实验,发现:数据处理深度系统性提升模型能力,是与规模扩展并列的关键维度;不同领域存在差异化的饱和动态,需采取比例调整或格式转变等自适应策略;组合平衡可实现定向强化而不引发性能崩溃;评估协议的选择深刻影响对预训练进展的理解。通过完整释放探索过程,我们使社区能够基于这些发现与系统方法论,积累预训练领域的科学知识。
原文摘要 · Abstract (English)
The foundational pretraining phase determines a model's capability ceiling, as post-training struggles to overcome capability foundations established during pretraining, yet it remains critically under-explored. This stems from a structural paradox: organizations with computational resources operate under commercial pressures that inhibit transparent disclosure, while academic institutions possess research freedom but lack pretraining-scale computational resources. daVinci-LLM occupies this unexplored intersection, combining industrial-scale resources with full research freedom to advance the science of pretraining. We adopt a fully-open paradigm that treats openness as scientific methodology, releasing complete data processing pipelines, full training processes, and systematic exploration results. Recognizing that the field lacks systematic methodology for data processing, we employ the Data Darwinism framework, a principled L0-L9 taxonomy from filtering to synthesis. We train a 3B-parameter model from random initialization across 8T tokens using a two-stage adaptive curriculum that progressively shifts from foundational capabilities to reasoning-intensive enhancement. Through 200+ controlled ablations, we establish that: processing depth systematically enhances capabilities, establishing it as a critical dimension alongside volume scaling; different domains exhibit distinct saturation dynamics, necessitating adaptive strategies from proportion adjustments to format shifts; compositional balance enables targeted intensification while preventing performance collapse; how evaluation protocol choices shape our understanding of pretraining progress. By releasing the complete exploration process, we enable the community to build upon our findings and systematic methodologies to form accumulative scientific knowledge in pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。