提出轻量方法识别广告检索模型的在线缩放规律,指导资源分配与模型设计。
Scaling Laws for Online Advertisement Retrieval
- 设计新离线指标与仿真算法,低成本估算线上收益与机器成本。
- 实验证明指标与线上收益相关性趋近1,跨多种模型架构有效。
- 可用于广告系统中资源受限的模型优化与多场景部署决策。
缩放定律是神经网络模型的重要特性,显著推动了大语言模型的发展。近期研究显示,缩放定律不仅适用于自然语言处理任务或Transformer架构,也适用于推荐系统等领域。然而,针对在线广告检索系统的缩放定律研究仍很缺乏,原因在于:1)在工业应用中,识别资源成本与线上收入之间的缩放关系需耗费大量时间和训练资源;2)不同系统设置差异大,导致缩放定律难以跨场景应用。为此,我们提出一种轻量级范式,用于识别检索模型的在线缩放规律,包含一个新颖的离线指标和一个离线仿真算法。我们在弱假设下证明该指标与线上收入的相关性渐近趋近于1,并通过实验验证其有效性。仿真算法可离线估计机器成本。基于此范式,我们几乎仅通过离线实验即可识别在线缩放规律,并快速估算给定模型配置下的机器成本与收益。进一步在真实广告系统中验证了主流模型架构(如Transformer、MLP、DSSM)均存在缩放规律。利用这些规律,我们展示了在收益-投资比约束下的模型设计及多场景资源分配的实际应用。据我们所知,这是首个研究在线广告检索系统中缩放规律识别与应用的工作。
原文摘要 · Abstract (English)
The scaling law is a notable property of neural network models and has significantly propelled the development of large language models. Scaling laws hold great promise in guiding model design and resource allocation. Recent research increasingly shows that scaling laws are not limited to NLP tasks or Transformer architectures; they also apply to domains such as recommendation. However, there is still a lack of literature on scaling law research in online advertisement retrieval systems. This may be because 1) identifying the scaling law for resource cost and online revenue is often expensive in both time and training resources for industrial applications, and 2) varying settings for different systems prevent the scaling law from being applied across various scenarios. To address these issues, we propose a lightweight paradigm to identify online scaling laws of retrieval models, incorporating a novel offline metric and an offline simulation algorithm. We prove that under mild assumptions, the correlation between the novel metric and online revenue asymptotically approaches 1 and empirically validates its effectiveness. The simulation algorithm can estimate the machine cost offline. Based on the lightweight paradigm, we can identify online scaling laws for retrieval models almost exclusively through offline experiments, and quickly estimate machine costs and revenues for given model configurations. We further validate the existence of scaling laws across mainstream model architectures (e.g., Transformer, MLP, and DSSM) in our real-world advertising system. With the identified scaling laws, we demonstrate practical applications for ROI-constrained model designing and multi-scenario resource allocation in the online advertising system. To the best of our knowledge, this is the first work to study identification and application of online scaling laws for online advertisement retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。