测试技术检索基准随时间变化的可靠性，发现多数查询仍有效。
Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks
- 对比2024与2025年两个时间点的技术语料库，分析其演化对检索的影响。
- 95%以上查询在新语料中仍有相关文档，仅1个查询失效。
- 模型排名变化极小，相关性高达0.978，适合长期评估使用。
信息检索（IR）基准通常遵循经典范式，依赖静态预设语料库。然而，技术语料库中的时间变化（如API废弃、代码重构）可能导致现有基准过时。本文研究了时间语料漂移对技术领域检索基准FreshStack的影响。我们对比了2024年10月与2025年10月的两个独立语料快照，聚焦于LangChain相关查询。分析显示，除1个查询外，其余所有2024年的查询在2025年语料中仍能被完全支持，因相关文档已迁移到竞争对手仓库（如LlamaIndex）。进一步比较检索模型在两个快照上的表现，发现模型排名仅有微小变动，且在Recall@50下相关性高达0.978（Kendall τ）。结果表明，即使语料随时间演进，重新评估的检索基准仍具可靠性。所有数据与工具已公开于https://github.com/fresh-stack/driftbench。
原文摘要 · Abstract (English)
Information retrieval (IR) benchmarks typically follow the Cranfield paradigm, relying on static and predefined corpora. However, temporal changes in technical corpora, such as API deprecations and code reorganizations, can render existing benchmarks stale. In our work, we investigate how temporal corpus drift affects FreshStack, a retrieval benchmark focused on technical domains. We examine two independent corpus snapshots of FreshStack from October 2024 and October 2025 to answer questions about LangChain. Our analysis shows that all but one query posed in 2024 remain fully supported by the 2025 corpus, as relevant documents "migrate" from LangChain to competitor repositories, such as LlamaIndex. Next, we compare the accuracy of retrieval models on both snapshots and observe only minor shifts in model rankings, with overall strong correlation of up to 0.978 Kendall $τ$ at Recall@50. These results suggest that retrieval benchmarks re-judged with evolving temporal corpora can remain reliable for retrieval evaluation. We publicly release all our artifacts at https://github.com/fresh-stack/driftbench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。