用合成故事数据训练语言模型，发现少量数据也能生成优质文本。
BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training
- 用TinyStories数据训练GPT-Neo，探索小样本下语言建模效果。
- 少于100M词数据即可生成高质量故事续写并掌握语言知识。
- 合成数据在多数情况下削弱模型理解能力，适合低资源场景研究。
我们参与了第二届BabyLM挑战赛的Strict和Strict-Small赛道，任务是在数据受限条件下实现高效预训练，模拟人类发展过程。为此，我们研究了使用新近推出的短故事数据集TinyStories进行合成故事数据对语言预训练的影响。首先，在不同数据量下训练GPT-Neo模型，发现即使仅使用不到100M词，模型仍能生成高质量、原创的故事续写，并具备显著的语言知识。随后，我们在结合了部分TinyStories、GPT-Neo生成的故事补全以及部分BabyLM数据的混合数据集上训练LTG-BERT编码器模型。实验表明，合成数据偶尔带来微弱提升，但整体对语言理解有负面影响。本工作为低资源环境下合成故事数据的使用提供了初步研究，强调其在数据受限场景中的潜在增广价值。相关模型与代码已公开发布于GitHub。
原文摘要 · Abstract (English)
We describe our contribution to the Strict and Strict-Small tracks of the 2nd iteration of the BabyLM Challenge. The shared task is centered around efficient pre-training given data constraints motivated by human development. In response, we study the effect of synthetic story data in language pre-training using TinyStories: a recently introduced dataset of short stories. Initially, we train GPT-Neo models on subsets of TinyStories, while varying the amount of available data. We find that, even with access to less than 100M words, the models are able to generate high-quality, original completions to a given story, and acquire substantial linguistic knowledge. To measure the effect of synthetic story data, we train LTG-BERT encoder models on a combined dataset of: a subset of TinyStories, story completions generated by GPT-Neo, and a subset of the BabyLM dataset. Our experimentation reveals that synthetic data can occasionally offer modest gains, but overall have a negative influence on linguistic understanding. Our work offers an initial study on synthesizing story data in low resource settings and underscores their potential for augmentation in data-constrained language modeling. We publicly release our models and implementation on our GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。