Kunlun提升推荐系统效率，让算力投入与性能增长更可预测。
Kunlun: Establishing Scaling Laws for Massive-Scale Recommendation Systems through Unified Architecture Design
- 统一架构设计优化底层计算与资源分配，提升模型效率。
- MFU从17%升至37%，算力利用率翻倍，性能增长更稳定。
- 已落地Meta广告系统，适合大规模推荐场景的工程师和研究者。
构建可预测的规模定律对大规模推荐系统的资源设计与分配至关重要。尽管大语言模型已有成熟规模定律，但处理用户历史与上下文特征的推荐系统仍面临挑战，主要源于低效模块导致的缩放效率差，表现为模型浮点运算利用率（MFU）低和资源分配不合理。本文提出Kunlun架构，通过低层优化如广义点积注意力（GDPA）、分层种子池化（HSP）、滑动窗口注意力，以及高层创新如计算跳过（CompSkip）和事件级个性化，系统性提升效率。在NVIDIA B200 GPU上，MFU由17%提升至37%，缩放效率较现有方法翻倍。该架构已部署于Meta广告核心模型，产生显著生产效益。
原文摘要 · Abstract (English)
Deriving predictable scaling laws that govern the relationship between model performance and computational investment is crucial for designing and allocating resources in massive-scale recommendation systems. While such laws are established for large language models, they remain challenging for recommendation systems, especially those processing both user history and context features. We identify poor scaling efficiency as the main barrier to predictable power-law scaling, stemming from inefficient modules with low Model FLOPs Utilization (MFU) and suboptimal resource allocation. We introduce Kunlun, a scalable architecture that systematically improves model efficiency and resource allocation. Our low-level optimizations include Generalized Dot-Product Attention (GDPA), Hierarchical Seed Pooling (HSP), and Sliding Window Attention. Our high-level innovations feature Computation Skip (CompSkip) and Event-level Personalization. These advances increase MFU from 17% to 37% on NVIDIA B200 GPUs and double scaling efficiency over state-of-the-art methods. Kunlun is now deployed in major Meta Ads models, delivering significant production impact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。