用量化压缩推荐系统嵌入层，50倍省内存且不降效果
A Universal Framework for Compressing Embeddings in CTR Prediction
- 先按热度加权正则化，再用对比学习统一量化码分布
- 在三个数据集上实现超50倍内存压缩，效果持平或更优
- 适合大规模推荐系统部署，尤其关注推理效率的团队
精准点击率（CTR）预测对在线广告和推荐系统至关重要。近年来深度学习虽提升了特征交互建模能力，但嵌入层优化常被忽视。嵌入表存储类别与序列特征，规模庞大，常超出GPU内存，需存于CPU，导致高内存占用和频繁的GPU-CPU数据传输，增加延迟。为此，我们提出一种模型无关的嵌入压缩框架（MEC），通过量化预训练嵌入实现压缩，同时保持推荐质量。方法分两阶段：首先采用热度加权正则化，平衡高频与低频特征的代码分布；随后引入对比学习机制，确保量化后代码分布均匀，提升嵌入区分度。在三个数据集上的实验表明，该方法可将内存使用减少超过50倍，同时维持或优于现有模型的推荐性能。代码已开源：https://github.com/USTC-StarTeam/MEC。
原文摘要 · Abstract (English)
Accurate click-through rate (CTR) prediction is vital for online advertising and recommendation systems. Recent deep learning advancements have improved the ability to capture feature interactions and understand user interests. However, optimizing the embedding layer often remains overlooked. Embedding tables, which represent categorical and sequential features, can become excessively large, surpassing GPU memory limits and necessitating storage in CPU memory. This results in high memory consumption and increased latency due to frequent GPU-CPU data transfers. To tackle these challenges, we introduce a Model-agnostic Embedding Compression (MEC) framework that compresses embedding tables by quantizing pre-trained embeddings, without sacrificing recommendation quality. Our approach consists of two stages: first, we apply popularity-weighted regularization to balance code distribution between high- and low-frequency features. Then, we integrate a contrastive learning mechanism to ensure a uniform distribution of quantized codes, enhancing the distinctiveness of embeddings. Experiments on three datasets reveal that our method reduces memory usage by over 50x while maintaining or improving recommendation performance compared to existing models. The implementation code is accessible in our project repository https://github.com/USTC-StarTeam/MEC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。