解决推荐系统超长用户行为序列训练的数据冗余问题。
Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale

- 按版本延迟生成序列,只存一份原始数据,按需重建。
- 部署后数据资源使用下降,序列长度可大幅提升,模型效果显著提升。
- 适合大规模推荐系统、多租户环境下的高效训练场景。
现代深度学习推荐模型(DLRMs)遵循序列长度的扩展规律,推动用户交互历史(UIH)向超长序列演进。然而,工业界标准的‘胖行’范式将序列预展开至每个训练样本,导致存储与I/O瓶颈:数据冗余在多租户环境下加剧,数据基础设施消耗超过GPU训练能力。本文提出一种‘版本化延迟物化’范式,将UIH仅以标准化、不可变形式存储一次,并在训练时通过轻量级版本指针即时重建。通过双轨协议确保流式与批处理训练中的在线到离线一致性,不可变存储层支持多维投影下推,适配异构模型租户。解耦数据预处理并结合流水线I/O预取与数据亲和性优化,掩盖了训练时序列重建的延迟,使训练吞吐量保持在GPU计算瓶颈。该系统已在生产级DLRM中部署,降低数据基础设施资源消耗的同时支持激进的序列长度扩展,带来显著模型质量提升,成为现代推荐模型架构(如HSTU和ULTRA-HSTU)的基础数据设施。
原文摘要 · Abstract (English)
Modern Deep Learning Recommendation Models (DLRMs) follow scaling laws with sequence length, driving the frontier toward ultra-long User Interaction History (UIH). However, the industry-standard "Fat Row" paradigm, which pre-materializes these sequences into every training example, creates a storage and I/O wall where data infrastructure usage exceeds GPU training capacity due to data redundancy that is amplified in multi-tenant environments where models with vastly different sequence length requirements share a union dataset. We present a \emph{versioned late materialization} paradigm that eliminates this redundancy by storing UIH once in a normalized, immutable tier and reconstructing sequences just-in-time during training via lightweight versioned pointers. The system ensures Online-to-Offline (O2O) consistency through a bifurcated protocol that prevents future leakage across both streaming and batch training, while a read-optimized immutable storage layer provides multi-dimensional projection pushdown for heterogeneous model tenants. Disaggregated data preprocessing with pipelined I/O prefetching and data-affinity optimizations masks the latency of training-time sequence reconstruction, keeping training throughput compute-bound by GPUs. Deployed on production DLRMs, the system reduces training data infrastructure resource usage while enabling aggressive sequence length scaling that delivers significant model quality gains, serving as the foundational data infrastructure for modern recommendation model architectures, including HSTU and ULTRA-HSTU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。