发现评测数据时间信号易受题目形式影响，可能误导模型污染判断。
Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- 用LLM改写题目会改变性能衰减模式，即使原始材料不变。
- 同一数据集上，填空题显示明显性能下降，改写后消失。
- 适合关注模型评测可靠性与污染检测方法的研究者。
大语言模型在截止日期后的性能衰减常被解释为基准数据污染的时序信号，即训练前公开的信息可能被记忆导致性能虚高。本文批判性检验这一观点，发现该时序信号对评测题目构造方式极为敏感，即使底层语料不变。具体而言，从相同文档中直接提取的填空题（cloze）与经由大语言模型转换的题目，会呈现显著不同的性能衰减模式。我们在报告明显性能衰减的LiveCodeBench等基准上验证此现象，发现仅通过大语言模型对问题进行转换，即可有效消除原本存在的时序信号。我们进一步通过影响函数分析提供机制解释。结果表明，性能衰减作为污染信号高度敏感，亟需更稳健的污染探测方法以实现可靠的模型评估。
原文摘要 · Abstract (English)
Post-cutoff performance decay of LLMs has been widely interpreted as a temporal signal for benchmark contamination, where public information released before the training cutoff may have been included into training corpora and inflated model performance by memorization. We critically examine this view and demonstrate that this temporal signal is highly sensitive to how benchmark questions are constructed, even if the underlying source material remains invariant. Specifically, we show that LLM-transformed questions can produce remarkably different temporal patterns compared to fill-in-the-blank (cloze) questions directly retrieved from the very same documents. We validate this effect on prior benchmarks that report clear post-cutoff decay (LiveCodeBench), and show that a simple LLM-driven transformation of the same problems can effectively remove the temporal pattern. We further provide a mechanistic understanding of this phenomenon using influence function analysis. Overall, our results suggest that post-cutoff performance decay is a sensitive contamination signal, motivating more robust contamination probes for reliable LLM evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。