数据越多,推荐效果越好,目前没看到性能饱和点。
The Unreasonable Effectiveness of Data for Recommender Systems
- 用9种数据量从10万到1亿次交互训练模型,评估效果变化。
- 在11个大规模数据集上,性能随数据量持续提升,未见拐点。
- 适合关注数据价值与推荐系统可扩展性的研究者和工程师。
在推荐系统中,收集、存储和处理大规模交互数据的成本越来越高,但增加数据是否仍有实际收益尚不明确。本文研究了离线推荐性能随训练数据规模增长的变化趋势,并检验是否存在性能饱和点。通过使用LensKit和RecBole两个工具包,构建可复现的评估流程,涵盖11个公开大型数据集(至少700万次交互),评估了10种工具-算法组合。采用分层用户抽样方法,分别在10万至1亿次交互的9个样本规模下训练模型,测量NDCG@10指标。结果表明,原始NDCG普遍随数据量上升,未观测到饱和点。对每组结果进行最小-最大归一化后,约75%的最大学样本点也达到了该组最佳性能。对每个数据组最后10%-30%的晚期斜率分析显示,四分位距始终非负,中位数接近1.0,进一步支持持续提升趋势。总体而言,在典型用户-物品交互数据上,传统推荐系统仍能从更多数据中获益;而较弱的缩放行为主要出现在非典型数据集及RecBole BPR算法的异常情况中。
原文摘要 · Abstract (English)
In recommender systems, collecting, storing, and processing large-scale interaction data is increasingly costly in terms of time, energy, and computation, yet it remains unclear when additional data stops providing meaningful gains. This paper investigates how offline recommendation performance evolves as the size of the training dataset increases and whether a saturation point can be observed. We implemented a reproducible Python evaluation workflow with two established toolkits, LensKit and RecBole, included 11 large public datasets with at least 7 million interactions, and evaluated 10 tool-algorithm combinations. Using absolute stratified user sampling, we trained models on nine sample sizes from 100,000 to 100,000,000 interactions and measured NDCG@10. Overall, raw NDCG usually increased with sample size, with no observable saturation point. To make result groups comparable, we applied min-max normalization within each group, revealing a clear positive trend in which around 75% of the points at the largest completed sample size also achieved the group's best observed performance. A late-stage slope analysis over the final 10-30% of each group further supported this upward trend: the interquartile range remained entirely non-negative with a median near 1.0. In summary, for traditional recommender systems on typical user-item interaction data, incorporating more training data remains primarily beneficial, while weaker scaling behavior is concentrated in atypical dataset cases and in the algorithmic outlier RecBole BPR under our setup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。