arXiv:2511.18805cs.IR2025-11被引 6

解决推荐系统中高基数特征的可扩展性与效率难题。

STORE: Semantic Tokenization, Orthogonal Rotation and Efficient Attention for Scaling Up Ranking Models

  • 将高基数特征分解为稳定语义令牌,缓解稀疏问题。
  • 在线上测试中提升点击率2.71%,AUC提高1.195%。
  • 适合大规模推荐系统开发者参考使用。

排序模型已成为现代个性化推荐系统的重要组成部分。然而,在处理高基数、异构且稀疏的特征空间时,仍面临模型可扩展性与效率的重大挑战。我们识别出两大瓶颈:(i) 表示瓶颈:受特征高基数和动态性影响,模型容量被迫集中在稀疏激活的嵌入层,导致低秩表示,引发‘单周期’和‘交互坍缩’现象,最终限制模型可扩展性;(ii) 计算瓶颈:将所有异构特征整合到统一模型中,导致特征令牌数量激增,使传统注意力机制计算开销巨大且易出现注意力分散。为突破这些障碍,我们提出STORE,一种基于三个核心创新的统一可扩展令牌化排序框架:(1) 语义令牌化通过将高基数稀疏特征分解为一组紧凑稳定的语义令牌,从根本上应对特征异构性和稀疏性;(2) 正交旋转变换用于旋转低基数静态特征张成的子空间,促进更高效、有效的特征交互;(3) 高效注意力机制通过过滤低贡献令牌,在保持模型精度的同时提升计算效率。在广泛的离线实验和线上A/B测试中,该框架持续提升预测精度(线上点击率提升2.71%,AUC提升1.195%)和训练效率(吞吐量提升1.84倍)。

原文摘要 · Abstract (English)

Ranking models have become an important part of modern personalized recommendation systems. However, significant challenges persist in handling high-cardinality, heterogeneous, and sparse feature spaces, particularly regarding model scalability and efficiency. We identify two key bottlenecks: (i) Representation Bottleneck: Driven by the high cardinality and dynamic nature of features, model capacity is forced into sparse-activated embedding layers, leading to low-rank representations. This, in turn, triggers phenomena like "One-Epoch" and "Interaction-Collapse," ultimately hindering model scalability.(ii) Computational Bottleneck: Integrating all heterogeneous features into a unified model triggers an explosion in the number of feature tokens, rendering traditional attention mechanisms computationally demanding and susceptible to attention dispersion. To dismantle these barriers, we introduce STORE, a unified and scalable token-based ranking framework built upon three core innovations: (1) Semantic Tokenization fundamentally tackles feature heterogeneity and sparsity by decomposing high-cardinality sparse features into a compact set of stable semantic tokens; and (2) Orthogonal Rotation Transformation is employed to rotate the subspace spanned by low-cardinality static features, which facilitates more efficient and effective feature interactions; and (3) Efficient attention that filters low-contributing tokens to improve computional efficiency while preserving model accuracy. Across extensive offline experiments and online A/B tests, our framework consistently improves prediction accuracy(online CTR by 2.71%, AUC by 1.195%) and training effeciency (1.84 throughput).

推荐系统排序模型高效注意力语义令牌

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。