让大模型按时间分段训练，避免未来信息泄露。
TiMoE: Time-Aware Mixture of Language Experts
- 用不同时期的文本分别训练多个专家，推理时只选过去时间窗口的专家。
- 在8个任务上减少15%的未来知识错误，性能接近单时段最佳专家。
- 适合需要时间敏感性的问答、新闻生成等场景。
大型语言模型通常基于固定时间点的网络数据训练，导致知识过时且存在时间泄露问题：可能依赖查询时间之后的信息。本文提出从零开始，在2013-2024年数据中按两年间隔划分的独立语料上预训练一系列GPT风格专家，并通过TiMoE（时间感知的专家混合）进行整合。推理时，TiMoE会屏蔽训练截止时间晚于查询时间戳的专家，将剩余专家的对数概率在共享空间合并，确保严格的因果有效性，同时保留多时期知识广度。我们还发布了TSQA基准，包含1万道题，每道题的选项明确标注为过去、未来或无关，可精细评估时间幻觉。在8个标准NLP任务及TSQA上的实验表明，协同优化的TiMoE变体性能达到甚至超过单一时间段最优专家，未来知识错误最多降低15%。结果表明，分段式预训练结合因果路由是使大模型保持时间一致性且不牺牲通用性能的有效方法。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) are typically trained on fixed snapshots of the web, which means that their knowledge becomes stale and their predictions risk temporal leakage: relying on information that lies in the future relative to a query. We tackle this problem by pre-training from scratch a set of GPT-style experts on disjoint two-year slices of a 2013-2024 corpus and combining them through TiMoE, a Time-aware Mixture of Language Experts. At inference time, TiMoE masks all experts whose training window ends after the query timestamp and merges the remaining log-probabilities in a shared space, guaranteeing strict causal validity while retaining the breadth of multi-period knowledge. We also release TSQA, a 10k-question benchmark whose alternatives are explicitly labelled as past, future or irrelevant, allowing fine-grained measurement of temporal hallucinations. Experiments on eight standard NLP tasks plus TSQA show that a co-adapted TiMoE variant matches or exceeds the best single-period expert and cuts future-knowledge errors by up to 15%. Our results demonstrate that modular, time-segmented pre-training paired with causal routing is a simple yet effective path toward LLMs that stay chronologically grounded without sacrificing general performance much. We open source our code at TiMoE (Github): https://github.com/epfml/TiMoE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。