针对长序列用户行为推荐,提出高效稀疏注意力模型SparseCTR。
Unleashing the Potential of Sparse Attention on Long-term Behaviors for CTR Prediction
- 按用户个性化分段行为序列,支持并行处理
- 三分支稀疏注意力捕捉全局、过渡与短期兴趣
- 在真实场景中提升点击率1.72%,具显著缩放效应
近年来,大语言模型的成功推动了推荐系统中规模定律的探索。然而,标准自注意力机制因计算复杂度高,难以在工业场景中部署用于建模长序列用户行为。尽管已有多种稀疏自注意力方法,但其不完全适用于推荐场景,因用户行为具有个性化和时序特性:不同用户行为模式各异且随时间演变,数据分布与其他领域差异显著。为此,我们提出SparseCTR,一种专为用户长期行为设计的高效有效模型。首先,采用个性化方式将行为序列分块,避免打断连续行为,支持序列并行处理。基于这些块,提出三分支稀疏自注意力机制,联合识别用户的全局兴趣、兴趣转移和短期兴趣。此外,设计复合相对时间编码,通过可学习的头特定偏置系数,更好捕捉行为间的顺序与周期关系。大量实验表明,SparseCTR不仅提升效率,还优于当前最优方法。更重要的是,其表现出明显的规模定律现象,在FLOPs跨三个数量级下仍保持性能提升。在线A/B测试中,点击率提升1.72%,千次展示收益提升1.41%。源码已开源:https://github.com/laiweijiang/SparseCTR。
原文摘要 · Abstract (English)
In recent years, the success of large language models (LLMs) has driven the exploration of scaling laws in recommender systems. However, models that demonstrate scaling laws are actually challenging to deploy in industrial settings for modeling long sequences of user behaviors, due to the high computational complexity of the standard self-attention mechanism. Despite various sparse self-attention mechanisms proposed in other fields, they are not fully suited for recommendation scenarios. This is because user behaviors exhibit personalization and temporal characteristics: different users have distinct behavior patterns, and these patterns change over time, with data from these users differing significantly from data in other fields in terms of distribution. To address these challenges, we propose SparseCTR, an efficient and effective model specifically designed for long-term behaviors of users. To be precise, we first segment behavior sequences into chunks in a personalized manner to avoid separating continuous behaviors and enable parallel processing of sequences. Based on these chunks, we propose a three-branch sparse self-attention mechanism to jointly identify users' global interests, interest transitions, and short-term interests. Furthermore, we design a composite relative temporal encoding via learnable, head-specific bias coefficients, better capturing sequential and periodic relationships among user behaviors. Extensive experimental results show that SparseCTR not only improves efficiency but also outperforms state-of-the-art methods. More importantly, it exhibits an obvious scaling law phenomenon, maintaining performance improvements across three orders of magnitude in FLOPs. In online A/B testing, SparseCTR increased CTR by 1.72\% and CPM by 1.41\%. Our source code is available at https://github.com/laiweijiang/SparseCTR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。