提升推荐系统对超长用户行为序列的建模能力，兼顾效果与效率。
LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders
- 引入全局令牌和轻量级合并模块，降低注意力计算复杂度。
- 在字节跳动广告与电商场景中，离线与线上指标均显著优于基线。
- 适合需要处理海量用户长序列数据的工业级推荐系统使用。
在工业级推荐系统中，建模超长用户行为序列对于捕捉长期与短期偏好至关重要。现有方案多依赖两阶段检索或间接建模范式，存在上下游不一致与计算效率低的问题。本文提出 LONGER，一种面向 GPU 高效推荐系统的长序列优化 Transformer。LONGER 引入 (i) 全局令牌机制以稳定长上下文注意力，(ii) 轻量级 InnerTransformers 与混合注意力策略的令牌合并模块，降低二次复杂度，(iii) 一系列工程优化，包括混合精度训练、激活重计算、KV 缓存服务及全同步模型训练与推理框架，实现稠密与稀疏参数的统一 GPU 更新。LONGER 在字节跳动的广告与电商业务中，持续优于强基线，在离线指标与线上 A/B 测试中表现优异。目前，LONGER 已在字节跳动超过 10 个核心场景全面部署，服务数十亿用户。
原文摘要 · Abstract (English)
Modeling ultra-long user behavior sequences is critical for capturing both long- and short-term preferences in industrial recommender systems. Existing solutions typically rely on two-stage retrieval or indirect modeling paradigms, incuring upstream-downstream inconsistency and computational inefficiency. In this paper, we present LONGER, a Long-sequence Optimized traNsformer for GPU-Efficient Recommenders. LONGER incorporates (i) a global token mechanism for stabilizing attention over long contexts, (ii) a token merge module with lightweight InnerTransformers and hybrid attention strategy to reduce quadratic complexity, and (iii) a series of engineering optimizations, including training with mixed-precision and activation recomputation, KV cache serving, and the fully synchronous model training and serving framework for unified GPU-based dense and sparse parameter updates. LONGER consistently outperforms strong baselines in both offline metrics and online A/B testing in both advertising and e-commerce services at ByteDance, validating its consistent effectiveness and industrial-level scaling laws. Currently, LONGER has been fully deployed at more than 10 influential scenarios at ByteDance, serving billion users.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。