权重衰减在推荐系统中实际是编码热门度信息,而非单纯正则化。
On the Role of Weight Decay in Collaborative Filtering: A Popularity Perspective
- 发现权重衰减主要将物品热度信息编码到嵌入向量大小中。
- 提出PRISM方法,使模型性能提升4.77%,训练时间减少38.48%。
- 适合想简化调参、缓解热门物品偏见的研究者和工程师。
协同过滤(CF)通过将历史用户-物品交互信息编码为稠密的ID嵌入表,支撑大规模推荐系统。随着嵌入表规模增大,闭式解不再可行,通常需采用小批量梯度下降训练。尽管损失函数设计已有大量研究,但一个核心组件——权重衰减——却常被忽视。高性能模型通常需精细调节权重衰减,无论使用何种损失函数,其必要性却未被充分理解。本文通过理论与实证分析发现,权重衰减的主要作用是将物品热度信息编码进嵌入向量的幅值中。此外,调节权重衰减可作为粗粒度非线性控制开关,影响对热门或冷门物品的偏好。基于此,我们提出PRISM(Popularity-awaRe Initialization Strategy for embedding Magnitudes),一种简单有效的嵌入幅值初始化策略,预先编码本应由权重衰减学习的热度信息,从而消除其依赖。实验表明,相比最先进训练策略,PRISM可提升性能达4.77%,训练时间减少38.48%。同时,我们参数化了PRISM以调节初始化强度,提供一种低成本且有意义的缓解热度偏差策略。
原文摘要 · Abstract (English)
Collaborative filtering (CF) enables large-scale recommendation systems by encoding information from historical user-item interactions into dense ID-embedding tables. However, as embedding tables grow, closed-form solutions become impractical, often necessitating the use of mini-batch gradient descent for training. Despite extensive work on designing loss functions to train CF models, we argue that one core component of these pipelines is heavily overlooked: weight decay. Attaining high-performing models typically requires careful tuning of weight decay, regardless of loss, yet its necessity is not well understood. In this work, we question why weight decay is crucial in CF pipelines and how it impacts training. Through theoretical and empirical analysis, we surprisingly uncover that weight decay's primary function is to encode popularity information into the magnitudes of the embedding vectors. Moreover, we find that tuning weight decay acts as a coarse, non-linear knob to influence preference towards popular or unpopular items. Based on these findings, we propose PRISM (Popularity-awaRe Initialization Strategy for embedding Magnitudes), a straightforward yet effective solution to simplify the training of high-performing CF models. PRISM pre-encodes the popularity information typically learned through weight decay, eliminating its necessity. Our experiments show that PRISM improves performance by up to 4.77% and reduces training times by 38.48%, compared to state-of-the-art training strategies. Additionally, we parameterize PRISM to modulate the initialization strength, offering a cost-effective and meaningful strategy to mitigate popularity bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。