用抽象数据预训练语言模型，显著提升推理能力并加速训练。
Procedural Pretraining: Warming Up Language Models with Abstract Data
- 用形式语言生成的程序化数据预训练模型，提升逻辑与结构理解力。
- 仅0.1%-0.3%的程序化数据即可使模型性能超越传统预训练，节省40%以上算力。
- 适用于希望高效训练、提升推理能力的研究者或工程团队。
直接在大规模网页语料上预训练语言模型是当前主流方法。本文提出一种替代方案：先用抽象结构化数据（如形式语言生成的程序化数据）预训练模型，以帮助后续学习丰富语义知识，类比人类先学逻辑与数学再进行高阶推理。我们聚焦于程序化数据，发现其可显著提升算法能力，例如在需要上下文回忆的任务中（Needle-in-a-haystack），准确率从10%跃升至98%（基于带括号的Dyck序列预训练）。进一步研究显示，即使仅用0.1%-0.3%的程序化数据预训练大模型（最大达1.3B参数），其性能也优于标准自然语言、代码及非正式数学数据（C4、CodeParrot、DeepMind-Math）的预训练方式。值得注意的是，该方法可在仅55/67/86%原始数据量下达到相同损失值，大幅降低计算开销。机制分析表明，程序化预训练在注意力和MLP层中引入了非平凡的结构，前者对代码等结构化任务尤为关键，后者则影响语言建模。最后，本文为融合多种程序化数据提供路径。结果表明，程序化预训练是一种简单轻量但高效的手段，可加速模型训练并提升性能，暗示将知识获取与推理能力解耦的潜力。
原文摘要 · Abstract (English)
Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to abstract structured data to ease the subsequent acquisition of rich semantic knowledge, much like humans learning simple logic and mathematics before higher reasoning. We focus on procedural data, generated by formal languages and other simple algorithms, as such abstract data. We first diagnose the algorithmic skills that different forms of procedural data can improve, often significantly. For example, the accuracy of context recall (Needle-in-a-haystack) jumps from 10 to 98% when a model is pretrained on Dyck sequences (balanced brackets). Second, we study how these gains are reflected in pretraining larger models (up to 1.3B). We find that front-loading as little as 0.1 to 0.3% procedural data significantly outperforms standard pretraining on natural language, code, and informal mathematics (C4, CodeParrot, and DeepMind-Math datasets). Notably, this also enables the models to reach the same loss value with only 55/67/86% of the original data and thus a comparable reduction in FLOPs. Third, we explore the mechanisms behind the benefits and find that procedural pretraining instills non-trivial structure in both attention and MLP layers. The former is particularly important for structured domains (e.g. code), and the latter for language. Finally, we lay a path for combining multiple forms of procedural data. Our results show that procedural pretraining is a simple, lightweight means of improving performance and accelerating language model pretraining, ultimately suggesting the promise of disentangling knowledge acquisition from reasoning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。