用每日新闻做持续评估，看大模型能否预测未来事件。
Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
- 用每日新闻自动生成问答对，动态测试模型预测能力。
- 模型性能随预训练数据过时而下降，时间越久准确率越低。
- 即使加了检索增强，仍难逃性能退化，适合长期跟踪模型演化。
现有大模型评估基准因新模型和训练数据的出现而迅速过时，且缺乏时间维度，无法衡量模型性能随时间的变化。为此，我们提出以未来事件预测作为持续评估方法，检验大模型的时间泛化与预测能力。我们的基准「Daily Oracle」从每日新闻中自动生成问题-答案对，要求模型预测“未来”事件的结果。结果显示，随着预训练数据过时，模型性能随时间持续下降。尽管检索增强生成（RAG）有潜力提升预测准确率，但性能退化趋势依然存在，凸显了持续更新模型的必要性。代码与数据已公开于 https://agenticlearning.ai/daily-oracle。
原文摘要 · Abstract (English)
Many existing evaluation benchmarks for Large Language Models (LLMs) quickly become outdated due to the emergence of new models and training data. These benchmarks also fall short in assessing how LLM performance changes over time, as they consist of a static set of questions without a temporal dimension. To address these limitations, we propose using future event prediction as a continuous evaluation method to assess LLMs' temporal generalization and forecasting abilities. Our benchmark, Daily Oracle, automatically generates question-answer (QA) pairs from daily news, challenging LLMs to predict "future" event outcomes. Our findings reveal that as pre-training data becomes outdated, LLM performance degrades over time. While Retrieval Augmented Generation (RAG) has the potential to enhance prediction accuracy, the performance degradation pattern persists, highlighting the need for continuous model updates. Code and data are available at https://agenticlearning.ai/daily-oracle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。