用语义分组取代随机哈希,提升推荐系统嵌入稳定性。
Enhancing Embedding Representation Stability in Recommendation Systems with Semantic ID
- 基于内容嵌入聚类生成语义碰撞的ID前缀ngram
- 在尾部物品建模上提升3.2%点击率,减少过拟合
- 适合高基数、动态增长的工业级推荐系统
在线内容激增给工业推荐系统的基于ID的模型带来挑战,包括极高的类别基数、不断增长的ID空间、高度偏斜的用户行为分布,以及因ID自然生命周期(如新ID出现与旧ID淘汰)导致的预测不稳定性。现有系统常采用随机哈希处理ID空间并控制模型参数(即嵌入表),但该方法引发多个ID共享同一嵌入的问题,导致数据污染、性能下降和嵌入表示不稳定。本文提出语义ID前缀ngram,一种新型标记参数化技术,通过基于内容嵌入的层次聚类创建语义有意义的碰撞,替代随机分配。实验表明,该方法不仅缓解嵌入不稳定性,显著改善尾部物品建模,降低过拟合,还减轻表示漂移。在基于注意力机制的上下文化用户历史模型中表现更优。该技术已集成至Meta广告排序系统,在线上部署中实现显著性能提升与预测稳定性增强。
原文摘要 · Abstract (English)
The exponential growth of online content has posed significant challenges to ID-based models in industrial recommendation systems, ranging from extremely high cardinality and dynamically growing ID space, to highly skewed engagement distributions, to prediction instability as a result of natural id life cycles (e.g, the birth of new IDs and retirement of old IDs). To address these issues, many systems rely on random hashing to handle the id space and control the corresponding model parameters (i.e embedding table). However, this approach introduces data pollution from multiple ids sharing the same embedding, leading to degraded model performance and embedding representation instability. This paper examines these challenges and introduces Semantic ID prefix ngram, a novel token parameterization technique that significantly improves the performance of the original Semantic ID. Semantic ID prefix ngram creates semantically meaningful collisions by hierarchically clustering items based on their content embeddings, as opposed to random assignments. Through extensive experimentation, we demonstrate that Semantic ID prefix ngram not only addresses embedding instability but also significantly improves tail id modeling, reduces overfitting, and mitigates representation shifts. We further highlight the advantages of Semantic ID prefix ngram in attention-based models that contextualize user histories, showing substantial performance improvements. We also report our experience of integrating Semantic ID into Meta production Ads Ranking system, leading to notable performance gains and enhanced prediction stability in live deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。