老基准过时了，用它评估大模型事实性会失真。
When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation
- 构建新检索管道与评估指标，量化基准随时间失效程度。
- 五大数据集超半数样本已过时，影响大模型评估可靠性。
- 适合关注评测可信度的研究者和开发者参考。
大型语言模型（LLMs）及其所处现实世界的快速演进，已超越广泛使用的静态评估基准的时效性，引发对其评估可靠性的担忧。尽管大量研究仍在依赖陈旧基准，但这些基准与真实世界事实及现代大模型之间的时序错位问题，以及其对事实性评估的影响仍未被充分探讨。为此，本文系统性地研究该问题，分析了五个流行的事实性基准与八款跨年发布的大型语言模型。我们设计了最新的事实检索流程与三项评估指标，以量化基准老化及其对模型评估的影响。实验结果与分析表明，广泛使用的一些基准中存在相当比例的过时样本，导致对大模型事实性的评估不可靠。我们希望本工作能为评估基准可靠性提供测试平台，并推动更多关于基准老化问题的研究。代码已开源：https://github.com/JiangXunyi/BenchAge。
原文摘要 · Abstract (English)
The rapid evolution of large language models (LLMs) and the real world has outpaced the static nature of widely used evaluation benchmarks, raising concerns about their reliability for evaluating LLM factuality. While substantial works continue to rely on the popular but old benchmarks, their temporal misalignment with real-world facts and modern LLMs, and their effects on LLM factuality evaluation remain underexplored. Therefore, in this work, we present a systematic investigation of this issue by examining five popular factuality benchmarks and eight LLMs released across different years. An up-to-date fact retrieval pipeline and three metrics are tailored to quantify benchmark aging and its impact on LLM factuality evaluation. Experimental results and analysis illustrate that a considerable portion of samples in the widely used factuality benchmarks are outdated, leading to unreliable assessments of LLM factuality. We hope our work can provide a testbed to assess the reliability of a benchmark for LLM factuality evaluation and inspire more research on the benchmark aging issue. Codes are available in https://github.com/JiangXunyi/BenchAge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。