LLM推荐系统评估可能因数据泄露而虚高，真实性能被严重误判。
Benchmark Leakage Trap: Can We Trust LLM-based Recommendation?
- 通过混合领域数据继续预训练，模拟模型数据泄露场景。
- 相关领域泄露导致性能虚高，无关领域泄露则降低准确率。
- 揭示评测结果不可靠根源，适合关注模型可信评估的研究者。
大型语言模型（LLMs）融入推荐系统带来了评估可靠性危机。本文首次识别并研究了其中被忽视的问题：基于LLM的推荐系统存在基准数据泄露现象。当模型在预训练或微调过程中接触并可能记忆基准数据集时，会导致性能指标人为虚高，无法反映真实能力。我们通过在包含域内和域外用户-物品交互的合成语料上进行持续预训练，模拟多种数据泄露场景。实验表明，数据泄露具有双重效应：若泄露数据与领域相关，会引发显著但虚假的性能提升，误导对模型能力的判断；若泄露数据无关，则通常降低推荐准确性，凸显该污染问题的复杂性和条件性。研究发现，数据泄露是影响基于LLM推荐系统真实性能的关键、此前未被考虑的因素。代码已开源：https://github.com/yusba1/LLMRec-Data-Leakage。
原文摘要 · Abstract (English)
The expanding integration of Large Language Models (LLMs) into recommender systems poses critical challenges to evaluation reliability. This paper identifies and investigates a previously overlooked issue: benchmark data leakage in LLM-based recommendation. This phenomenon occurs when LLMs are exposed to and potentially memorize benchmark datasets during pre-training or fine-tuning, leading to artificially inflated performance metrics that fail to reflect true model performance. To validate this phenomenon, we simulate diverse data leakage scenarios by conducting continued pre-training of foundation models on strategically blended corpora, which include user-item interactions from both in-domain and out-of-domain sources. Our experiments reveal a dual-effect of data leakage: when the leaked data is domain-relevant, it induces substantial but spurious performance gains, misleadingly exaggerating the model's capability. In contrast, domain-irrelevant leakage typically degrades recommendation accuracy, highlighting the complex and contingent nature of this contamination. Our findings reveal that data leakage acts as a critical, previously unaccounted-for factor in LLM-based recommendation, which could impact the true model performance. We release our code at https://github.com/yusba1/LLMRec-Data-Leakage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。