解决10万级用户行为序列建模的高效计算难题,实现工业级端到端推荐。
SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching
- 用可学习原型压缩长序列,生成固定大小的用户表征
- 结合短期10K与长期压缩表征,提升推荐精度
- 通过低秩缓存和流水线优化,降低存储通信开销
建模长期用户行为是序列推荐和百亿规模工业推荐系统的核心,但生产环境中的排序模型受限于延迟、内存、通信和训练吞吐量。在10万级序列下,挑战不仅来自注意力复杂度:原始序列特征需在训练和在线服务中反复存储、传输和处理。现有方法如历史截断、多阶段行为检索、压缩终身历史或训练短/推理长外推,均削弱了端到端优化或仍存在显著长度相关成本。我们提出SequenceO1,一个端到端的超长用户行为序列建模框架,在抖音全流量部署,支持长达10万次交互的历史。SequenceO1采用先压缩后推理的设计:其Sketch Attention(SA)利用可学习原型和原型归一化,将原始历史压缩为固定大小、目标无关的用户表征;目标条件化的堆叠目标-历史交叉注意力(STCA)则建模互补时间尺度——近期10K尾部用于短期兴趣,紧凑的sketch用于长期偏好。为使训练和推理可行,SequenceO1结合低秩用户表征缓存、多请求用户级批处理、流水线提升和融合FlashSA内核,分摊跨目标、训练实例和连续请求的特征存储、通信与计算成本。生产实验显示一致的离线与在线提升,且紧凑缓存的sketch保留了直接扩展端到端序列排序至10万级的大部分收益。这些结果为高效注意力、序列压缩及可扩展长序列与长上下文推荐系统提供了实用的模型-系统协同方案。
原文摘要 · Abstract (English)
Modeling long-term user behavior is central to sequential recommendation and billion-scale industrial recommender systems, yet production ranking models operate under strict latency, memory, communication, and training-throughput constraints. At the 100K scale, the challenge extends beyond attention complexity: raw sequence features must be stored, transferred, and repeatedly processed during training and online serving. Existing approaches based on history truncation, multi-stage behavior retrieval, compressed lifelong histories, or train-short/infer-long extrapolation either weaken end-to-end optimization or retain substantial length-dependent cost. We present SequenceO1, an end-to-end framework for ultra-long user behavior sequence modeling, deployed at full traffic on Douyin with histories of up to 100K interactions. SequenceO1 follows a compress-then-reason design. Its Sketch Attention (SA) uses learnable prototypes and prototype-wise normalization to compress the raw history into a fixed-size, target-agnostic user representation. Target-conditioned Stacked Target-to-History Cross Attention (STCA) then models complementary time scales: a recent 10K suffix for short-term interests and the compact sketch for long-term preferences. To make training and inference practical, SequenceO1 combines low-rank user representation caching, multi-request user-level batching, pipeline lift, and a fused FlashSA kernel to amortize feature storage, communication, and computation across targets, training instances, and consecutive requests. Production experiments show consistent offline and online gains, while the compact cached sketch retains most of the benefit of directly scaling end-to-end sequence ranking to 100K. These results provide a practical model-system approach to efficient attention, sequence compression, and scalable long-sequence and long-context recommendation systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。