在内存有限时，应优先增大批次而非负样本数以加快推荐模型训练。
Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training
- 固定内存下，理论分析表明大批次小负样本更优。
- 实测显示该配置收敛更快且推荐效果更好。
- 适合资源受限场景下的推荐系统训练调参。
大规模神经推荐系统通常采用对完整物品词表的softmax交叉熵目标函数进行训练。当可能的物品数量 $K$ 很大时，最终分类层占据主要内存，需为一批 $n$ 个样本存储 $O(nK)$ 的logits和梯度。采样softmax通过将目标限制在仅 $k \\<\< K$ 个候选负样本上，将内存降至 $O(nk)$。然而，在固定预算 $B = n k$ 下，仍不清楚应优先增大批次 $n$ 还是负样本数 $k$。本文在固定内存约束下分析采样softmax训练，基于标准光滑性和方差假设，理论表明最优收敛速度来自 $n \sim B, k \sim 1$ 的分配。因此，建议在计算约束下尽可能包含更多样本。该理论经合成数据及四个真实序列推荐基准（含MovieLens-20M）验证，所提配置在相同内存下收敛更快、最终推荐质量更高。研究为推荐系统训练中的内存配置提供了理论与实证基础。代码、可复现材料及所有绘图脚本见 https://anonymous.4open.science/r/LimitedMemoryRule-BBFB。
原文摘要 · Abstract (English)
Large-scale neural recommender systems are typically trained with a softmax cross-entropy objective over the full item vocabulary. For a typical large number of possible items $K$, the final classification layer dominates memory, requiring $O(nK)$ logits and gradients to materialize for a batch of $n$ examples. Sampled softmax reduces this cost by restricting the objective to only $k \ll K$ candidate negative items, resulting in an $O(nk)$ memory. However, for a fixed budget $B = n k$, it remains unclear whether one should prioritize larger batches or the inclusion of more negative items. We address this question by analyzing sampled-softmax training under a fixed memory constraint. Under standard smoothness and variance assumptions, our theoretical evidence suggests that the fastest convergence arises from an $ n \sim B, k \sim 1$ allocation. So, an actionable rule is to include as many objects as possible given computational constraints. Our theory is supported by controlled synthetic and synthetic and four real sequential recommendation benchmarks, including MovieLens-20M. The suggested configuration achieve faster convergence and better final recommendation quality than imbalanced alternatives within the same memory constraint. These findings provide a theoretical and empirical foundation for configuring memory during the training of recommender systems. Code, reproducibility materials, and all scripts for generating figures are available at https://anonymous.4open.science/r/LimitedMemoryRule-BBFB
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。