用低秩分解优化注意力，让推荐系统处理超长行为序列
SOLAR: SVD-Optimized Lifelong Attention for Recommendation
- 基于SVD重构注意力机制，保持softmax且降低计算复杂度
- 支持十万个行为序列与数千候选商品的端到端推荐
- 在快手线上场景提升视频播放量0.68%，适合长序列推荐
注意力机制是Transformer的核心，但其$O(N^2 d)$的时间和内存开销使长序列建模成本高昂，常需截断或启发式处理。线性注意力通过核特征映射重排计算，将复杂度降至$O(N d^2)$，但损失了softmax机制并改变注意力分布。在推荐系统中，矩阵的低秩结构并非例外，而是表征学习的默认归纳偏置，尤其体现在用户行为序列建模中。我们提出SVD-Attention，理论上对低秩矩阵无损，保留softmax的同时将注意力复杂度从$O(N^2 d)$降至$O(Ndr)$。基于此，我们构建SOLAR（SVD-Optimized Lifelong Attention for Recommendation），一个支持十万级行为序列与数千项候选集端到端建模的序列框架，无需任何过滤。在快手线上推荐场景中，SOLAR带来0.68%的视频播放量提升，并带动其他业务指标改善。
原文摘要 · Abstract (English)
Attention mechanism remains the defining operator in Transformers since it provides expressive global credit assignment, yet its $O(N^2 d)$ time and memory cost in sequence length $N$ makes long-context modeling expensive and often forces truncation or other heuristics. Linear attention reduces complexity to $O(N d^2)$ by reordering computation through kernel feature maps, but this reformulation drops the softmax mechanism and shifts the attention score distribution. In recommender systems, low-rank structure in matrices is not a rare case, but rather the default inductive bias in its representation learning, particularly explicit in the user behavior sequence modeling. Leveraging this structure, we introduce SVD-Attention, which is theoretically lossless on low-rank matrices and preserves softmax while reducing attention complexity from $O(N^2 d)$ to $O(Ndr)$. With SVD-Attention, we propose SOLAR, SVD-Optimized Lifelong Attention for Recommendation, a sequence modeling framework that supports behavior sequences of ten-thousand scale and candidate sets of several thousand items in cascading process without any filtering. In Kuaishou's online recommendation scenario, SOLAR delivers a 0.68\% Video Views gain together with additional business metrics improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。