测试大模型对时间上下文变化的敏感度，发现其常识判断常出错。
Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations
- 设计新数据集TimeStress，测试模型在不同时间上下文下的事实判断能力。
- 最优模型仅11%的事实能完全正确区分时间上下文，错误率虽低但致命。
- 揭示当前大模型在时间推理上的根本缺陷，适合关注常识与可靠性研究者。
本文研究语言模型（LMs）在事实知识中对时间上下文变化的鲁棒性。通过要求模型区分正确与错误的时间上下文，考察其是否能准确关联事实与有效时间段。评估从两个维度展开：错误上下文与有效期的距离、上下文粒度。为此构建了名为TimeStress的数据集，用于评估18种不同LMs。结果表明，最优模型仅在11%的已研究事实中实现完全正确区分，且错误虽罕见，但对人类而言属于明显可避免的失误。该研究凸显当前语言模型在时间表征方面的显著局限。
原文摘要 · Abstract (English)
This paper explores the robustness of language models (LMs) to variations in the temporal context within factual knowledge. It examines whether LMs can correctly associate a temporal context with a past fact valid over a defined period, by asking them to differentiate correct from incorrect contexts. The LMs' ability to distinguish is analyzed along two dimensions: the distance of the incorrect context from the validity period and the granularity of the context. To this end, a dataset called TimeStress is introduced, enabling the evaluation of 18 diverse LMs. Results reveal that the best LM achieves a perfect distinction for only 11% of the studied facts, with errors, certainly rare, but critical that humans would not make. This work highlights the limitations of current LMs in temporal representation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。