用合成数据提升语言模型预训练效果，让模型学会文档间的深层关联。
Synthetic bootstrapped pretraining
- 先学文档间关系，再生成新文本用于联合训练
- 3B/6B模型在1T tokens上训练，性能提升达60%的理论上限
- 生成内容不止改写，而是抽象概念后重构叙事
我们提出合成自举预训练（SBP），一种语言模型预训练方法：先从预训练数据中学习文档间的关联关系，再利用该关系生成海量新语料进行联合训练。传统预训练仅关注单文档内词元的因果关联，无法高效建模文档间的丰富可学习关联。我们在计算量匹配的设置下，从零开始预训练30亿和60亿参数模型，使用最多1万亿词的语料。结果表明，SBP持续优于强基线重复策略，性能提升达到拥有20倍更多唯一数据的最优基准的60%。定性分析显示，合成文档并非简单改写，而是先抽象原始材料的核心概念，再在此基础上构建新叙述。此外，SBP具有自然的贝叶斯解释：合成器隐式学习到相关文档共享的潜在概念。
原文摘要 · Abstract (English)
We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure that first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training. While the standard pretraining teaches LMs to learn causal correlations among tokens within a single document, it is not designed to efficiently model the rich, learnable inter-document correlations that can potentially lead to better performance. We validate SBP by designing a compute-matched pretraining setup and pretrain a 3B-parameter and a 6B-parameter model on up to 1T tokens from scratch. We find SBP consistently improves upon a strong repetition baseline and delivers up to 60% of performance improvement attainable by an oracle upper bound with access to 20x more unique data. Qualitative analysis reveals that the synthesized documents go beyond mere paraphrases -- SBP first abstracts a core concept from the seed material and then crafts a new narration on top of it. Besides strong empirical performance, SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns to abstract the latent concepts shared between related documents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。