提出线性注意力模型FuXi-Linear,提升长序列推荐效率与精度。
FuXi-Linear: Unleashing the Power of Linear Attention in Long-term Time-aware Sequential Recommendation
- 分离时序与语义信号,独立建模行为周期性。
- 在千级序列上实现推荐性能超越基线,推理速度提升21倍。
- 适合处理超长用户行为序列的工业级推荐场景。
现代推荐系统依赖二次复杂度的注意力机制,难以处理长用户序列且推理缓慢。尽管线性注意力是潜在解决方案,但现有研究面临三大挑战:(1)时序信号常被忽略或粗暴耦合,导致时序与语义信号相互干扰,并忽略行为周期性;(2)现有线性框架提供的位置信息不足;(3)主要聚焦短序列与浅层结构。为此,我们提出FuXi-Linear,一种线性复杂度的长序列推荐模型。其核心包含:(1)时序保留通道,通过时序数据独立计算周期性注意力权重,避免信号干扰;(2)线性位置通道,利用可学习核在低复杂度下整合位置信息。此外,我们发现FuXi-Linear在千级序列尺度下具备稳健的幂律缩放特性,这是此前线性推荐研究未探索的。在数千个标记的序列上进行大量实验表明,该模型在推荐质量上优于当前最优方法,预填充阶段提速最高达10倍,解码阶段提速最高达21倍。代码已开源:https://github.com/USTC-StarTeam/fuxi-linear。
原文摘要 · Abstract (English)
Modern recommendation systems primarily rely on attention mechanisms with quadratic complexity, which limits their ability to handle long user sequences and slows down inference. While linear attention is a promising alternative, existing research faces three critical challenges: (1) temporal signals are often overlooked or integrated via naive coupling that causes mutual interference between temporal and semantic signals while neglecting behavioral periodicity; (2) insufficient positional information provided by existing linear frameworks; and (3) a primary focus on short sequences and shallow architectures. To address these issues, we propose FuXi-Linear, a linear-complexity model designed for efficient long-sequence recommendation. Our approach introduces two key components: (1) a Temporal Retention Channel that independently computes periodic attention weights using temporal data, preventing crosstalk between temporal and semantic signals; (2) a Linear Positional Channel that integrates positional information through learnable kernels within linear complexity. Moreover, we demonstrate that FuXi-Linear exhibits a robust power-law scaling property at a thousand-length scale, a characteristic largely unexplored in prior linear recommendation studies. Extensive experiments on sequences of several thousand tokens demonstrate that FuXi-Linear outperforms state-of-the-art models in recommendation quality, while achieving up to 10$\times$ speedup in the prefill stage and up to 21$\times$ speedup in the decode stage compared to competitive baselines. Our code has been released in a public repository https://github.com/USTC-StarTeam/fuxi-linear.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。