数据剪枝让推荐系统评估结果虚高,真实性能反而下降。
Dataset Pruning in RecSys and ML: Best Practice or Mal-Practice?
- 对用户交互数低于阈值的数据进行剪枝,影响数据分布与算法表现。
- 剪枝后仅保留2%用户,传统算法在测试集上性能显著下降。
- 适合关注评估可信度的研究者,警惕被剪枝数据误导的结论。
推荐系统研究中的离线评估严重依赖数据集,而许多数据集(如MovieLens)经过剪枝处理,即移除交互次数少于特定阈值的用户。本文分析了五种基准数据集在未剪枝状态及五种剪枝级别(5、10、20、50、100)下的结构与分布特性,并在每种核心集上训练和测试了十一个代表性算法。为检验剪枝是否导致性能虚高,还评估了在剪枝训练集上训练、但在未剪枝测试集上测试的模型表现。结果显示,常见剪枝策略极具选择性,部分数据集中仅保留2%原始用户;当训练与测试均在剪枝数据上时,传统算法的nDCG@10得分较高;但换到未剪枝测试集后,这一优势几乎消失。所有算法在未剪枝测试集上的性能随剪枝程度增加而下降,表明数据缩减严重影响推荐算法的真实表现。
原文摘要 · Abstract (English)
Offline evaluations in recommender system research depend heavily on datasets, many of which are pruned, such as the widely used MovieLens collections. This thesis examines the impact of data pruning - specifically, removing users with fewer than a specified number of interactions - on both dataset characteristics and algorithm performance. Five benchmark datasets were analysed in both their unpruned form and at five successive pruning levels (5, 10, 20, 50, 100). For each coreset, we examined structural and distributional characteristics and trained and tested eleven representative algorithms. To further assess if pruned datasets lead to artificially inflated performance results, we also evaluated models trained on the pruned train sets but tested on unpruned data. Results show that commonly applied core pruning can be highly selective, leaving as little as 2% of the original users in some datasets. Traditional algorithms achieved higher nDCG@10 scores when both training and testing on pruned data; however, this advantage largely disappeared when evaluated on unpruned test sets. Across all algorithms, performance declined with increasing pruning levels when tested on unpruned data, highlighting the impact of dataset reduction on the performance of recommender algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。