一键搜索推荐系统嵌入维度,兼顾效果与内存。
AdaS&S: a One-Shot Supernet Approach for Automatic Embedding Size Search in Deep Recommender System
- 构建包含多种嵌入的超网络,分离训练与搜索过程。
- 提升AUC约0.3%,减少20%参数量,结果更稳定。
- 适合资源受限场景,支持自动适配内存约束。
深度学习推荐模型(DLRM)使用嵌入层表示各类类别特征。传统DLRM对所有特征采用统一嵌入大小,导致性能不佳且参数冗余。因此,大量自动嵌入尺寸搜索(AES)研究致力于获取混合嵌入尺寸以提升模型性能。然而,现有方法难以同时解决三大挑战:(1)嵌入尺寸搜索结果不稳定;(2)使用AES结果后的推荐效果不理想;(3)嵌入内存成本不可控。为此,我们提出一种新型的一次性AES框架AdaS&S,其核心是在一个包含多种候选嵌入的超网络中进行架构搜索。该框架分为两阶段:第一阶段,将训练参数与嵌入尺寸搜索解耦,提出自适应采样方法,构建训练充分的超网络,从而提升搜索结果稳定性;第二阶段,基于前期训练好的超网络设计强化学习搜索流程,以优化推荐效果,并引入资源竞争惩罚项,使搜索能适配特定资源约束。我们在多个公开数据集上进行了大量实验,结果表明AdaS&S在提升约0.3% AUC的同时,可节省约20%模型参数。实证分析显示,其搜索结果稳定性显著优于其他方法。
原文摘要 · Abstract (English)
Deep Learning Recommendation Model(DLRM)s utilize the embedding layer to represent various categorical features. Traditional DLRMs adopt unified embedding size for all features, leading to suboptimal performance and redundant parameters. Thus, lots of Automatic Embedding size Search (AES) works focus on obtaining mixed embedding sizes with strong model performance. However, previous AES works can hardly address several challenges together: (1) The search results of embedding sizes are unstable; (2) Recommendation effect with AES results is unsatisfactory; (3) Memory cost of embeddings is uncontrollable. To address these challenges, we propose a novel one-shot AES framework called AdaS&S, in which a supernet encompassing various candidate embeddings is built and AES is performed as searching network architectures within it. Our framework contains two main stages: In the first stage, we decouple training parameters from searching embedding sizes, and propose the Adaptive Sampling method to yield a well-trained supernet, which further helps to produce stable AES results. In the second stage, to obtain embedding sizes that benefits the model effect, we design a reinforcement learning search process which utilizes the supernet trained previously. Meanwhile, to adapt searching to specific resource constraint, we introduce the resource competition penalty to balance the model effectiveness and memory cost of embeddings. We conduct extensive experiments on public datasets to show the superiority of AdaS&S. Our method could improve AUC by about 0.3% while saving about 20% of model parameters. Empirical analysis also shows that the stability of searching results in AdaS&S significantly exceeds other methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。