通过两阶段对比学习预训练，解决推荐系统单轮训练难题
Taming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-training
- 用小模型+对比损失做两阶段预训练，扩大数据覆盖
- 预训练可多轮迭代不过拟合，提升下游任务泛化能力
- 已在Pinterest上线，显著提升全站用户参与度
基于ID的嵌入广泛应用于超大规模在线推荐系统，但其易受长尾数据分布影响而过拟合，常导致训练仅限于单轮，即“单轮现象”。为突破此限制，研究聚焦于第一轮内加速收敛或增强特征稀疏性。本文提出一种新颖的两阶段训练策略：先用小型模型结合对比损失进行预训练，实现嵌入系统的更广数据覆盖。离线实验表明，预训练阶段可进行多轮训练而不引发过拟合，所获嵌入在微调至复杂下游推荐任务时表现出更强的在线泛化能力。该方案已在Pinterest真实流量中部署，带来显著的全站用户参与度提升。
原文摘要 · Abstract (English)
ID-based embeddings are widely used in web-scale online recommendation systems. However, their susceptibility to overfitting, particularly due to the long-tail nature of data distributions, often limits training to a single epoch, a phenomenon known as the "one-epoch problem." This challenge has driven research efforts to optimize performance within the first epoch by enhancing convergence speed or feature sparsity. In this study, we introduce a novel two-stage training strategy that incorporates a pre-training phase using a minimal model with contrastive loss, enabling broader data coverage for the embedding system. Our offline experiments demonstrate that multi-epoch training during the pre-training phase does not lead to overfitting, and the resulting embeddings improve online generalization when fine-tuned for more complex downstream recommendation tasks. We deployed the proposed system in live traffic at Pinterest, achieving significant site-wide engagement gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。