分离注意力与表示的嵌入,提升长序列推荐精度与效率
Long-Sequence Recommendation Models Need Decoupled Embeddings
- 用两套独立嵌入分别处理注意力和表示,避免相互干扰
- 在公开数据集上最高提升AUC 0.9%,腾讯广告平台效果显著
- 可降低注意力嵌入维度,搜索速度加快50%仍保持性能
长期用户行为序列对捕捉用户兴趣和预测响应至关重要。现有长序列推荐模型普遍采用两阶段范式:第一阶段通过注意力机制从原始长序列中筛选相关行为,第二阶段将这些行为与目标项聚合,构建用于预测的判别性表示。本文首次揭示并分析了现有模型中的一个被忽视的问题:单一嵌入集难以同时学习注意力与表示,导致两者间存在干扰。初步尝试使用线性投影等通用方法(源自自然语言处理)解决此问题,但效果不佳,凸显推荐模型的独特挑战。为此,我们提出解耦注意力与表示嵌入(DARE)模型,通过初始化并独立学习两套嵌入表,彻底解耦注意力与表示过程。大量实验与分析表明,DARE能更准确地搜索相关行为,在公共数据集上最高提升AUC 0.9%,并在腾讯广告平台实现显著改进。此外,解耦嵌入空间允许降低注意力嵌入维度,使搜索过程提速50%而性能影响较小,支持更高效、高性能的线上服务。PyTorch代码及模型分析已开源:https://github.com/thuml/DARE。
原文摘要 · Abstract (English)
Lifelong user behavior sequences are crucial for capturing user interests and predicting user responses in modern recommendation systems. A two-stage paradigm is typically adopted to handle these long sequences: a subset of relevant behaviors is first searched from the original long sequences via an attention mechanism in the first stage and then aggregated with the target item to construct a discriminative representation for prediction in the second stage. In this work, we identify and characterize, for the first time, a neglected deficiency in existing long-sequence recommendation models: a single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes. Initial attempts to address this issue with some common methods (e.g., linear projections -- a technique borrowed from language processing) proved ineffective, shedding light on the unique challenges of recommendation models. To overcome this, we propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are initialized and learned separately to fully decouple attention and representation. Extensive experiments and analysis demonstrate that DARE provides more accurate searches of correlated behaviors and outperforms baselines with AUC gains up to 0.9% on public datasets and notable improvements on Tencent's advertising platform. Furthermore, decoupling embedding spaces allows us to reduce the attention embedding dimension and accelerate the search procedure by 50% without significant performance impact, enabling more efficient, high-performance online serving. Code in PyTorch for experiments, including model analysis, is available at https://github.com/thuml/DARE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。