按时间顺序训练能让大模型掌握更实时的常识知识。
Understanding Data Temporality Impact on Large Language Models Pre-training
- 用时间有序的网页数据训练模型，而非打乱顺序
- 顺序训练的模型知识更新更快，旧数据重复率更低
- 适合关注模型时效性与持续学习的研究者
大语言模型通常在打乱顺序的语料上训练，导致知识停留在训练时点，时间定位不清晰。本文研究预训练中数据排序对时序性知识获取的影响。主要贡献有二：一是构建涵盖7000多个时间锚定问题的基准测试集及评估协议，可分析模型对事实与其时间背景的匹配能力；二是将60亿参数模型在时间有序的Common Crawl快照上进行预训练，并与标准打乱训练对比。结果表明，顺序训练的模型在通用语言理解与常识上与打乱训练相当，但在时序知识上更准确、更及时。时间有序训练提升了事实的新鲜度，而打乱训练则在较旧数据上表现更好，可能因旧数据重复率更高。相关代码、检查点与数据集已开源，为大模型持续学习研究奠定基础。
原文摘要 · Abstract (English)
Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at train time and whose temporal grounding remains poorly understood. In this work, we study the impact of pre-training dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pre-training yields improved factual freshness, while shuffled pre-training peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our code at https://github.com/kyutai-labs/kairos , checkpoints, and datasets at https://huggingface.co/collections/kyutai/kairos provide a foundation for future research on continual learning for LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。