RankMixer提升推荐模型效率,100倍扩参下延迟不变,实测用户活跃度提升0.3%。
RankMixer: Scaling Up Ranking Models in Industrial Recommenders
- 用多头令牌混合替代注意力,兼顾高并行与低延迟
- 模型参数扩至百亿级,推理延迟几乎不变,算力利用率从4.5%提至45%
- 适配工业级高并发场景,已在推荐与广告中验证效果
大型语言模型的进展激发了推荐系统规模化的兴趣,但仍有两大现实挑战:工业级推荐系统需满足严格延迟约束和高每秒查询率(QPS)要求;现有多数人工设计的特征交叉模块源自CPU时代,难以利用现代GPU,导致模型浮点运算利用率(MFU)低、可扩展性差。我们提出RankMixer,一种面向硬件优化的统一可扩展特征交互架构。该架构保留Transformer的高并行性,以多头令牌混合模块取代二次复杂度自注意力,显著提升效率。同时通过逐令牌前馈网络,兼顾不同特征子空间建模与跨特征空间交互。进一步通过稀疏门控专家(Sparse-MoE)扩展至百亿参数规模,采用动态路由策略缓解专家训练不均问题。在万亿级生产数据集上的实验表明,替换原有低效手工模块后,模型MFU从4.5%提升至45%,参数量扩大100倍,推理延迟基本不变。在线A/B测试验证其在推荐与广告两大核心场景的通用性。最终上线10亿参数的稠密版RankMixer,服务全流量无额外成本,使用户活跃天数提升0.3%,应用内总使用时长增加1.08%。
原文摘要 · Abstract (English)
Recent progress on large language models (LLMs) has spurred interest in scaling up recommendation systems, yet two practical obstacles remain. First, training and serving cost on industrial Recommenders must respect strict latency bounds and high QPS demands. Second, most human-designed feature-crossing modules in ranking models were inherited from the CPU era and fail to exploit modern GPUs, resulting in low Model Flops Utilization (MFU) and poor scalability. We introduce RankMixer, a hardware-aware model design tailored towards a unified and scalable feature-interaction architecture. RankMixer retains the transformer's high parallelism while replacing quadratic self-attention with multi-head token mixing module for higher efficiency. Besides, RankMixer maintains both the modeling for distinct feature subspaces and cross-feature-space interactions with Per-token FFNs. We further extend it to one billion parameters with a Sparse-MoE variant for higher ROI. A dynamic routing strategy is adapted to address the inadequacy and imbalance of experts training. Experiments show RankMixer's superior scaling abilities on a trillion-scale production dataset. By replacing previously diverse handcrafted low-MFU modules with RankMixer, we boost the model MFU from 4.5\% to 45\%, and scale our ranking model parameters by 100x while maintaining roughly the same inference latency. We verify RankMixer's universality with online A/B tests across two core application scenarios (Recommendation and Advertisement). Finally, we launch 1B Dense-Parameters RankMixer for full traffic serving without increasing the serving cost, which improves user active days by 0.3\% and total in-app usage duration by 1.08\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。