用提示让大模型‘倒退’到过去知识状态，测试其遗忘能力。
Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs
- 通过提示模拟早于预训练数据的截止时间，测试模型遗忘效果。
- 直接提问后验信息时模型能有效忽略，但关联信息仍会泄露。
- 适合研究模型记忆与推理边界、评估时序预测可信度的研究者。
大型语言模型广泛用于时间预测任务，但其依赖预训练数据可能导致知识污染：在预截止日期的数据上表现准确，可能源于记忆而非推理，从而高估模型泛化能力。随着基于提示的遗忘技术兴起，一个自然问题是：能否通过提示让模型模拟更早的知识截止？本文构建三个评估数据集，测试模型在（1）直接事实知识、（2）语义演变、（3）因果关联知识上的遗忘能力。结果表明，当直接询问截止后信息时，提示式模拟知识截止有效；但若所问内容与被遗忘信息存在因果关联，模型仍会泄露知识。该发现强调了在时间预测任务中采用更严格评估设置的必要性。完整数据集与代码已开源于 https://github.com/gxx27/time_unlearn。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are widely used for temporal prediction, but their reliance on pretraining data raises contamination concerns, as accurate predictions on pre-cutoff test data may reflect memorization rather than reasoning, leading to an overestimation of their generalization capability. With the recent emergence of prompting-based unlearning techniques, a natural question arises: Can LLMs be prompted to simulate an earlier knowledge cutoff? In this work, we investigate the capability of prompting to simulate earlier knowledge cutoff in LLMs. We construct three evaluation datasets to assess the extent to which LLMs can forget (1) direct factual knowledge, (2) semantic shifts, and (3) causally related knowledge. Results demonstrate that while prompt-based simulated knowledge cutoffs show effectiveness when directly queried with the information after that date, they struggle to induce forgetting when the forgotten content is not directly asked but causally related to the query. These findings highlight the need for more rigorous evaluation settings when applying LLMs for temporal prediction tasks. The full dataset and evaluation code are available at https://github.com/gxx27/time_unlearn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。