构建动态知识问答基准,测试模型对新知识的检索与推理能力。
LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
- 通过对比维基数据快照差异自动生成问答题
- 新知识测试中模型表现明显下降,多跳查询差距最显著
- 适合评估大模型在实时知识更新下的真实推理能力
现有大语言模型问答评测多依赖静态基准,易奖励记忆而忽视检索作用,无法反映知识动态性。我们提出LiveSearchBench,一种从近期知识更新自动构建检索依赖型基准的流水线方法。该方法计算维基数据连续快照间的差异,筛选高质量候选三元组,并在三个推理难度层级上生成自然语言问题,每个问题均通过SPARQL验证保证答案唯一且可验证。整个流程完全自动化、可扩展,支持持续生成时序锚定的评测基准。实验显示,当面对预训练后新增的事实时,模型性能显著下降,尤其在多跳查询中差距突出。尽管检索增强方法和更大规模指令微调模型带来部分提升,仍无法弥合知识时效性差距。LiveSearchBench旨在推动评估从静态记忆转向需实时检索与推理的任务,为长期系统性评估大模型在动态知识下的表现奠定基础。
原文摘要 · Abstract (English)
Evaluating large language models (LLMs) on question answering often relies on static benchmarks that reward memorization and understate the role of retrieval, failing to capture the dynamic nature of world knowledge. We present LiveSearchBench, an automated pipeline for constructing retrieval-dependent benchmarks from recent knowledge updates. Our method computes deltas between successive Wikidata snapshots, filters candidate triples for quality, and synthesizes natural-language questions at three levels of reasoning difficulty, each guaranteed to admit a unique, verifiable answer through SPARQL validation. The pipeline is fully automated, scalable across time, and minimizes human intervention, enabling continual regeneration of temporally grounded benchmarks. Experiments show a pronounced performance drop when models confront facts that post-date pretraining, with the gap most salient on multi-hop queries. Retrieval augmented methods and larger, instruction-tuned models provide partial gains but fail to close this recency gap. By design, LiveSearchBench shifts evaluation from static memorization toward tasks that require up-to-date retrieval and reasoning, offering a foundation for systematic, long-term assessment of LLMs under evolving knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。