构建大规模时间持续预训练基准，评估模型如何在新数据中学习而不遗忘旧知识。
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
- 基于114个Common Crawl数据集构建超大规模时间持续预训练数据集。
- 自回归元调度+固定比例旧数据重放，计算量减少2.6倍仍保持与重训相当效果。
- 通用网页数据需重放旧数据防遗忘，特定领域则影响较小，策略应差异化。
基于历史网络数据训练的大语言模型不可避免地会过时。本文研究了新数据出现后对大模型的评估策略与更新方法。我们引入一个基于114个Common Crawl（CC）数据快照的超大规模时间持续预训练数据集，规模远超以往基准。同时设计跨时间分层的评估方案，涵盖通用CC数据及特定领域（维基百科、StackExchange、代码文档），以检验不同持续学习方法在吸收新数据的同时保留旧知识的能力。实验表明，在通用CC数据上，采用自回归元调度并结合固定比例旧数据重放，可在仅需2.6倍计算量的情况下达到与从头重训相当的保留损失；但重放对通用网页数据至关重要以避免遗忘，而在特定领域其作用减弱，最佳平衡策略因领域而异。
原文摘要 · Abstract (English)
Large Language Models (LLMs) trained on historical web data inevitably become outdated. We investigate evaluation strategies and update methods for LLMs as new data becomes available. We introduce a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Common Crawl (CC) - orders of magnitude larger than previous continual language modeling benchmarks. We also design time-stratified evaluations across both general CC data and specific domains (Wikipedia, StackExchange, and code documentation) to assess how well various continual learning methods adapt to new data while retaining past knowledge. Our findings demonstrate that, on general CC data, autoregressive meta-schedules combined with a fixed-ratio replay of older data can achieve comparable held-out loss to re-training from scratch, while requiring significantly less computation (2.6x). However, the optimal balance between incorporating new data and replaying old data differs as replay is crucial to avoid forgetting on generic web data but less so on specific domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。