为长序列推荐设计内存感知压缩基准,解决大规模系统内存开销问题
MALLOC: Benchmarking the Memory-aware Long Sequence Compression for Large Sequential Recommendation
- 构建面向长序列推荐的内存感知压缩评估框架
- 在真实规模下验证多种内存管理策略的压缩效果与效率
- 适合关注推荐系统可扩展性与内存优化的研究者
模型性能随数据量和模型容量增长的缩放定律推动了推荐系统向更大规模发展。然而,大规模推荐系统面临显著更高的计算成本,尤其在用户行为序列存在长依赖时。现有方法通常预存每位用户的中间状态以降低二次计算开销,但仅将内存视为加速介质,未充分考虑其空间开销。在拥有数十亿用户、每人上千次交互的现实场景中,状态存储需海量内存。尽管大语言模型中已有多种内存压缩策略,但大多未在推荐任务上评估。为此,我们提出 MALLOC,一个面向大规模序列推荐的内存感知长序列压缩综合基准。MALLOC 系统性地梳理并分类适用于推荐系统的内存管理技术,将其集成至前沿推荐模型中,构建可复现、易访问的评估平台。通过在准确性、效率与复杂度上的广泛实验,验证了 MALLOC 在推动大规模推荐发展中的整体可靠性。代码已公开于 https://anonymous.4open.science/r/MALLOC。
原文摘要 · Abstract (English)
The scaling law, which indicates that model performance improves with increasing dataset and model capacity, has fueled a growing trend in expanding recommendation models in both industry and academia. However, the advent of large-scale recommenders also brings significantly higher computational costs, particularly under the long-sequence dependencies inherent in the user intent of recommendation systems. Current approaches often rely on pre-storing the intermediate states of the past behavior for each user, thereby reducing the quadratic re-computation cost for the following requests. Despite their effectiveness, these methods often treat memory merely as a medium for acceleration, without adequately considering the space overhead it introduces. This presents a critical challenge in real-world recommendation systems with billions of users, each of whom might initiate thousands of interactions and require massive memory for state storage. Fortunately, there have been several memory management strategies examined for compression in LLM, while most have not been evaluated on the recommendation task. To mitigate this gap, we introduce MALLOC, a comprehensive benchmark for memory-aware long sequence compression. MALLOC presents a comprehensive investigation and systematic classification of memory management techniques applicable to large sequential recommendations. These techniques are integrated into state-of-the-art recommenders, enabling a reproducible and accessible evaluation platform. Through extensive experiments across accuracy, efficiency, and complexity, we demonstrate the holistic reliability of MALLOC in advancing large-scale recommendation. Code is available at https://anonymous.4open.science/r/MALLOC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。