arXiv:2506.13315cs.IR2025-06

提出新型线性注意力模型,高效建模用户长期行为偏好。

Gated Rotary-Enhanced Linear Attention with Rank Modulation for Long-term Sequential Recommendation

  • 用旋转位置编码提升长序列依赖建模效率
  • 引入自适应秩调节器,平衡长期与短期兴趣
  • 在多个数据集上表现最优,内存开销低

在序列推荐系统中,Transformer 模型虽性能出色,但因点积注意力的二次复杂度,在处理长序列时面临计算与内存瓶颈。线性注意力通过映射函数近似点积注意力,实现线性复杂度,但现有方法存在三方面局限:1)使用可学习位置编码,增加长序列计算开销;2)受低秩缺陷限制,难以捕捉用户细粒度局部偏好(如短期兴趣爆发);3)混淆临时行为与长期稳定兴趣,导致推荐效果模糊。为此,我们提出 Gated Rotary-Enhanced Linear Attention(RecGRELA)模型。首先设计旋转增强线性注意力(RELA)模块,利用旋转位置编码高效建模用户历史信息中的长程依赖;其次引入自适应秩调节器,包含秩增强分支以显式注入局部令牌混合,并通过门控选择器动态平衡长期偏好与短期兴趣。在四个公开基准数据集上的实验表明,RecGRELA 在性能上优于基于 RNN、Transformer 及 Mamba 的现有 SRS 模型,同时保持较低内存开销。

原文摘要 · Abstract (English)

In Sequential Recommendation Systems (SRSs), Transformer models have demonstrated remarkable performance but face computational and memory cost challenges, especially when modeling long-term user behavior sequences. Due to its quadratic complexity, the dot-product attention mechanism in Transformers becomes expensive for processing long sequences. By approximating the dot-product attention using elaborate mapping functions, linear attention provides a more efficient option with linear complexity. However, existing linear attention methods face three limitations: 1) they often use learnable position encodings, which incur extra computational costs in long-term sequence scenarios, 2) limited by the low-rank deficiency, they may not sufficiently account for user's fine-grained local preferences (short-lived burst of interest), and 3) they try to capture some temporary activities, but often confuse these with stable and long-term interests. This can result in unclear or less effective recommendations. To remedy these drawbacks, we propose a long-term sequential Recommendation model with Gated Rotary Enhanced Linear Attention (RecGRELA). Specifically, we first propose a Rotary-Enhanced Linear Attention (RELA) module to efficiently model long-range dependency within the user's historical information using rotary position encodings. Then, to address the low-rank deficiency of linear attention, we introduce an Adaptive Rank Modulator. It incorporates a rank augmentation branch to explicitly inject local token mixing and a Gated Rank Selector to dynamically balance stable long-term preferences and transient short-term interests. Experimental results on four public benchmark datasets show that our RecGRELA achieves state-of-the-art performance compared with existing SRSs based on Recurrent Neural Networks, Transformer, and Mamba while keeping low memory overhead.

序列推荐线性注意力长序列建模旋转编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。