用随机生成数据预训练模型，可实现零样本上下文学习。
Universal pre-training by iterated random computation
- 用随机计算生成数据进行模型预训练
- 预训练后模型在多种数据集上实现零样本学习，且规模越大效果越好
- 适合想提升模型泛化能力的研究者
我们研究了利用随机生成数据进行模型预训练的可行性。从算法复杂性角度出发，基于近期研究表明序列模型可逼近索洛莫诺夫归纳（Solomonoff induction），我们推导出相似但互补的理论结果。实验表明，合成数据可用于模型预训练，即使这些数据从未被模型见过。我们复现了早期成果：此类训练的模型在多种数据集上展现出零样本上下文学习能力，且性能随模型规模增大而提升。进一步扩展至真实世界数据，发现预训练后微调可加速收敛并提升泛化能力。
原文摘要 · Abstract (English)
We investigate the use of randomly generated data for the sake of pre-training a model. We justify this approach theoretically from the perspective of algorithmic complexity, building on recent research that shows that sequence models can be trained to approximate Solomonoff induction. We derive similar, but complementary theoretical results. We show empirically that synthetically generated data can be used to pre-train a model before the data is seen. We replicate earlier results that models trained this way show zero-shot in-context learning across a variety of datasets, and that this performance improves with scale. We extend earlier results to real-world data, and show that finetuning a model after pre-training offers faster convergence and better generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。