提出新数据集,帮大模型判断问题该不该用最新信息
How often do Answers Change? Estimating Recency Requirements in Question Answering
- 按答案更新频率和上下文依赖性分类问题
- 4031个问题标注了时效性与稳定性标签
- 发现上下文影响时效性的题更难,越频繁更新越难答
大型语言模型在回答时敏问题时常依赖过时知识,导致自信但错误的回答。由于缺乏是否需要最新信息的明确信号,模型难以决定何时检索外部证据、如何处理过时事实,以及如何根据有效性排序答案。现有基准要么定期刷新答案,要么依赖固定模板,无法反映答案实际更新频率或问题本身对时效性的要求。为此,我们提出一个时效性-稳定性分类体系,按答案变化频率及是否受上下文影响对问题进行分类。基于此体系,我们构建了包含4,031个开放域问题的RecencyQA数据集,每个问题均标注了时效性和稳定性标签。通过人工评估与实证分析,我们发现非平稳问题(即上下文影响时效性要求的问题)对大模型更具挑战性,且更新频率越高,难度越大。显式建模时效性与上下文依赖性,使我们能超越简单的“新鲜/陈旧”二元判断,实现更细粒度的时间推理评测,并为开发具备时效感知与上下文敏感能力的问答系统提供基础。
原文摘要 · Abstract (English)
Large language models (LLMs) often rely on outdated knowledge when answering time-sensitive questions, leading to confident yet incorrect responses. Without explicit signals indicating whether up-to-date information is required, models struggle to decide when to retrieve external evidence, how to reason about stale facts, and how to rank answers by their validity. Existing benchmarks either periodically refresh answers or rely on fixed templates, but they do not reflect on how frequently answers change or whether a question inherently requires up-to-date information. To address this gap, we introduce a recency-stationarity taxonomy that categorizes questions by how often their answers change and whether this change frequency is time-invariant or context-dependent. Building on this taxonomy, we present RecencyQA, a dataset of 4,031 open-domain questions annotated with recency and stationarity labels. Through human evaluation and empirical analysis, we show that non-stationary questions, i.e., those where context changes the recency requirement, are significantly more challenging for LLMs, with difficulty increasing as update frequency rises. By explicitly modeling recency and context dependence, RecencyQA enables fine-grained benchmarking and analysis of temporal reasoning beyond binary notions of freshness, and provides a foundation for developing recency-aware and context-sensitive question answering systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。