通过构建可扩展的点击率模型,实现线上性能显著提升。
Exploring Scaling Laws of CTR Model for Online Performance Improvement
- 提出新型可扩展点击率模型SUAN,融合序列与非序列特征建模。
- 轻量版LightSUAN经知识蒸馏后性能超越高阶原模型,推理时间不变。
- 已在真实线上服务部署,点击率提升2.81%,适合大规模在线应用。
点击率(CTR)模型在众多个性化在线服务中对改善用户体验和提升业务收入至关重要。然而,现有CTR模型普遍面临性能瓶颈。受大语言模型缩放定律启发,本文提出一种新范式:先构建精度随模型规模和数据量可扩展的CTR模型,再将其知识蒸馏至轻量级模型以支持线上部署。为此,我们设计了名为SUAN(Stacked Unified Attention Network)的模型,其核心为统一行为编码器UAB,能同时建模序列与非序列特征,并从多角度衡量用户行为重要性。堆叠多个UAB实现高阶配置,满足性能扩展需求。为兼顾高性能与低延迟,我们引入稀疏自注意力与并行推理策略,得到轻量版LightSUAN;并通过在线知识蒸馏,以高阶SUAN为教师训练低阶LightSUAN。实验表明,SUAN在模型规模与数据量跨越三个数量级下仍保持缩放规律,且蒸馏后的LightSUAN性能优于高一阶的SUAN。更重要的是,该模型已成功接入线上服务,使点击率提升2.81%,每千次展示收益(CPM)提升1.69%,平均推理时间仍可接受。代码已开源。
原文摘要 · Abstract (English)
CTR models play a vital role in improving user experience and boosting business revenue in many online personalized services. However, current CTR models generally encounter bottlenecks in performance improvement. Inspired by the scaling law phenomenon of LLMs, we propose a new paradigm for improving CTR predictions: first, constructing a CTR model with accuracy scalable to the model grade and data size, and then distilling the knowledge implied in this model into its lightweight model that can serve online users. To put it into practice, we construct a CTR model named SUAN (Stacked Unified Attention Network). In SUAN, we propose the UAB as a behavior sequence encoder. A single UAB unifies the modeling of the sequential and non-sequential features and also measures the importance of each user behavior feature from multiple perspectives. Stacked UABs elevate the configuration to a high grade, paving the way for performance improvement. In order to benefit from the high performance of the high-grade SUAN and avoid the disadvantage of its long inference time, we modify the SUAN with sparse self-attention and parallel inference strategies to form LightSUAN, and then adopt online distillation to train the low-grade LightSUAN, taking a high-grade SUAN as a teacher. The distilled LightSUAN has superior performance but the same inference time as the LightSUAN, making it well-suited for online deployment. Experimental results show that SUAN performs exceptionally well and holds the scaling laws spanning three orders of magnitude in model grade and data size, and the distilled LightSUAN outperforms the SUAN configured with one grade higher. More importantly, the distilled LightSUAN has been integrated into an online service, increasing the CTR by 2.81% and CPM by 1.69% while keeping the average inference time acceptable. Our source code is available at https://github.com/laiweijiang/SUAN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。