将抖音用户行为序列建模扩展至10000级,保持低延迟高效推荐。
Make It Long, Keep It Fast: End-to-End 10K Long User Behavior Sequence Modeling for Billion-Scale Douyin Recommendation
- 用目标-历史交叉注意力替代自注意力,复杂度从二次降为线性。
- 通过请求级批处理共享用户编码,降低存储与计算开销。
- 短序列训练、长序列推理,实现无额外成本的超长序列泛化。
抖音等短视频推荐系统需在不突破延迟和成本预算的前提下,利用极长的用户行为历史。本文提出一个端到端工业级推荐系统,可将用户行为序列建模扩展至10000长度。首先,提出堆叠的目标-历史交叉注意力(STCA),以目标对历史的交叉注意力替代历史自注意力,使复杂度从二次降至线性,支持超长序列的端到端训练。其次,设计请求级批处理(RLB),将同一用户/请求的多个目标聚合,共享用户侧编码,显著降低序列相关的存储、通信与计算开销,且不改变学习目标。第三,采用长度外推训练策略——在较短窗口上训练,推理时使用更长序列——使模型无需额外训练成本即可泛化至10000级历史。离线与在线实验均显示,随着历史长度与模型容量增加,性能持续稳定提升,符合大语言模型的缩放规律。该系统已全流量部署于抖音,显著提升关键互动指标,同时满足生产延迟要求,为端到端超长序列推荐提供了可行路径。
原文摘要 · Abstract (English)
Short-video recommenders such as Douyin must exploit extremely long user behavior histories without breaking latency or cost budgets. We present an end-to-end industrial recommender system that scales long-sequence recommendation modeling to 10K-length histories in production. First, we introduce Stacked Target-to-History Cross Attention (STCA), which replaces history self-attention with stacked cross-attention from the target to the history, reducing complexity from quadratic to linear in sequence length and enabling efficient end-to-end training over long user behavior sequences. Second, we propose Request Level Batching (RLB), a user-centric batching scheme that aggregates multiple targets for the same user/request to share the user-side encoding, substantially lowering sequence-related storage, communication, and compute without changing the learning objective. Third, we design a length-extrapolative training strategy -- train on shorter windows, infer on much longer ones -- so the model generalizes to 10K-scale histories without additional training cost. Across offline and online experiments, we observe predictable, monotonic gains as we scale history length and model capacity, mirroring the scaling law behavior observed in large language models. Deployed at full traffic on Douyin, our system delivers significant improvements on key engagement metrics while meeting production latency, demonstrating a practical path to scaling end-to-end ultra-long sequence recommendation to the 10K regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。