用分层低秩专家模型高效提升点击率预测性能
Hierarchical LoRA MoE for Efficient CTR Model Scaling
- 分层路由+低秩专家，实现参数高效水平与垂直扩展
- 相比基线模型，AUC提升0.20%，计算量减少18.5%
- 适合大规模推荐系统，追求高性价比模型扩容的团队
深度模型推动了点击率（CTR）预测的重大进展。虽然通过堆叠层数提升模型表达能力的纵向扩展有效，但逐层计算限制了效率。相反，通过专家混合（MoE）实现横向扩展可在并行中激活少量专家，实现高效扩展，但扁平的MoE结构难以捕捉推荐任务中的层次结构。为突破投入产出比边界，本文探索两种方向的互补优势，提出HiLoMoE——一种分层低秩专家混合框架，实现参数高效的全局扩展。具体地，HiLoMoE采用轻量级秩-1专家实现参数高效的横向扩展，并通过多层分层路由实现组合多样化的专家配置。不同于传统堆叠方式，其基于前层得分进行路由而非输出，使所有层可并行执行。一个三阶段训练框架保障优化稳定性和专家多样性。在四个公开数据集上的实验表明，相比非MoE基线，HiLoMoE平均提升0.20% AUC，FLOPs降低18.5%。
原文摘要 · Abstract (English)
Deep models have driven significant advances in click-through rate (CTR) prediction. While vertical scaling via layer stacking improves model expressiveness, the layer-by-layer sequential computation poses challenges to efficient scaling. Conversely, horizontal scaling through Mixture of Experts (MoE) achieves efficient scaling by activating a small subset of experts in parallel, but flat MoE layers may struggle to capture the hierarchical structure inherent in recommendation tasks. To push the Return-On-Investment (ROI) boundary, we explore the complementary strengths of both directions and propose HiLoMoE, a hierarchical LoRA MoE framework that enables holistic scaling in a parameter-efficient manner. Specifically, HiLoMoE employs lightweight rank-1 experts for parameter-efficient horizontal scaling, and stacks multiple MoE layers with hierarchical routing to enable combinatorially diverse expert compositions. Unlike conventional stacking, HiLoMoE routes based on prior layer scores rather than outputs, allowing all layers to execute in parallel. A principled three-stage training framework ensures stable optimization and expert diversity. Experiments on four public datasets show that HiLoMoE achieving better performance-efficiency tradeoff, achieving an average AUC improvement of 0.20\% in AUC and 18.5\% reduction in FLOPs compared to the non-MoE baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。