解决大品类推荐中交叉熵损失的内存瓶颈,提升效率且不降效果。
Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs
- 用最大内积搜索筛选关键物品,只计算高信息量项的softmax。
- 在多个数据集上峰值显存降低100倍,指标持平或更优。
- 适合大规模推荐系统、大模型训练等需要高效损失计算的场景。
可扩展性问题是生产化现代推荐系统的关键挑战。即使轻量级架构也可能因中间计算导致高计算负载,限制其在真实场景的应用。全量交叉熵(CE)损失虽能带来最优推荐质量,但在处理大品类目录时会引发过度的GPU内存占用。本文提出一种适用于序列学习场景的新型可扩展交叉熵(SCE)损失函数,对大规模目录数据集近似CE损失,在不牺牲推荐质量的前提下显著提升时间效率与内存使用效率。不同于传统负采样方法,本方法采用选择性GPU高效计算策略,聚焦于最可能为误报的目录元素,通过最大内积搜索在模型输出子集上近似softmax分布。多数据集实验表明,SCE可将峰值内存使用量最多降低100倍,同时保持或超越基线指标。该方法也为大规模语言模型等领域的扩展提供了新思路。
原文摘要 · Abstract (English)
Scalability issue plays a crucial role in productionizing modern recommender systems. Even lightweight architectures may suffer from high computational overload due to intermediate calculations, limiting their practicality in real-world applications. Specifically, applying full Cross-Entropy (CE) loss often yields state-of-the-art performance in terms of recommendations quality. Still, it suffers from excessive GPU memory utilization when dealing with large item catalogs. This paper introduces a novel Scalable Cross-Entropy (SCE) loss function in the sequential learning setup. It approximates the CE loss for datasets with large-size catalogs, enhancing both time efficiency and memory usage without compromising recommendations quality. Unlike traditional negative sampling methods, our approach utilizes a selective GPU-efficient computation strategy, focusing on the most informative elements of the catalog, particularly those most likely to be false positives. This is achieved by approximating the softmax distribution over a subset of the model outputs through the maximum inner product search. Experimental results on multiple datasets demonstrate the effectiveness of SCE in reducing peak memory usage by a factor of up to 100 compared to the alternatives, retaining or even exceeding their metrics values. The proposed approach also opens new perspectives for large-scale developments in different domains, such as large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。