通过随机分块与全局融合，提升推荐系统深层表征能力
RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems
- 用随机分块稀疏特征+多嵌入机制增强表征多样性
- 在微信视频号等场景实现3.4%~4.8%的成交额提升
- 适合大规模广告推荐系统优化表征容量的工程实践
推荐系统的缩放定律日益得到验证，基于MetaFormer的架构在增加模型深度、隐藏维度和用户行为序列长度时表现持续提升。然而，表征能力是否随参数增长而线性扩展仍不清楚。先前对RankMixer的研究发现，标记表示的有效秩在各层间呈现阻尼振荡趋势，随深度增加并未持续上升，甚至在深层出现退化。针对此现象，我们提出RankUp架构，通过稀疏特征的随机排列分割、多嵌入范式、全局标记整合及交叉预训练嵌入标记，缓解表征坍缩，提升表达能力。RankUp已在微信视频号、公众号和朋友圈大规模生产环境部署，分别带来3.41%、4.81%和2.12%的成交额提升。
原文摘要 · Abstract (English)
The scaling laws for recommender systems have been increasingly validated, where MetaFormer-based architectures consistently benefit from increased model depth, hidden dimensionality, and user behavior sequence length. However, whether representation capacity scales proportionally with parameter growth remains unexplored. Prior studies on RankMixer reveal that the effective rank of token representations exhibits a damped oscillatory trajectory across layers, failing to increase consistently with depth and even degrading in deeper layers. Motivated by this observation, we propose RankUp, an architecture designed to mitigate representation collapse and enhance expressive capacity through randomized permutation splitting over sparse features, a multi-embedding paradigm, global token integration and crossed pretrained embedding tokens. RankUp has been fully deployed in large-scale production across Weixin Video Accounts, Official Accounts and Moments, yielding GMV improvements of 3.41%, 4.81% and 2.12%, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。