arXiv:2605.23191cs.LGcs.IR2026-05KDD被引 1

提出RankElastor缓解推荐模型嵌入坍缩,提升可扩展性。

Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation

论文配图:Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation
图 1 · 摘自论文原文
  • 用参数化全混合与GLU改进前馈层,增强表示谱鲁棒性。
  • 在工业级数据集上显著降低嵌入坍缩,性能持续提升。
  • 适合大规模推荐系统研发者关注模型可扩展性优化。

推荐模型的扩展是推荐系统的核心挑战。近期,RankMixer通过统一的标记表示,在标记混合与逐标记前馈网络(P-FFNs)间交替,实现了可扩展性能。然而,RankMixer存在嵌入坍缩问题,导致学习到的表示有效秩过低,限制了表达能力并浪费扩展后的表示空间。通过实证分析与理论洞察,我们发现刚性的标记混合与P-FFN模块是该现象的主要原因,二者共同引发有效秩演化中的阻尼振荡轨迹。为此,我们提出新架构RankElastor,可生成谱鲁棒表示并提供坍缩缓解的理论保证。RankElastor引入两项改进:(i) 参数化全混合,实现更具表现力的标记混合与谱鲁棒性;(ii) GLU改进型P-FFNs,通过类似GLU的前馈结构稳定表示谱。在大规模工业数据集上的大量实验表明,RankElastor能持续提升推荐性能,缓解嵌入坍缩,并表现出稳健的扩展行为。代码已开源于https://github.com/vasile-paskardlgm/RankElastor。

原文摘要 · Abstract (English)

Scaling recommendation models is a central challenge in recommender systems. Recently, RankMixer has emerged as an effective solution, operating on a unified token representation and alternating between token mixing and per-token feedforward networks (P-FFNs) to achieve scalable performance. However, RankMixer suffers from \textit{embedding collapse}, where learned representations have low effective rank, limiting expressivity and underutilizing the expanded representation space. Through empirical analysis and theoretical insights, we identify rigid token mixing and P-FFN modules as the primary causes of this phenomenon, jointly inducing a \textbf{damped oscillatory trajectory} in effective-rank evolution across layers. To address it, we propose RankElastor, a novel architecture that produces spectrum-robust representations with provable collapse mitigation. RankElastor introduces two components: (i) \textbf{parameterized full mixing}, which enables expressive token mixing with improved spectral robustness; and (ii) \textbf{GLU-improved P-FFNs}, which stabilize representation spectra through GLU-style FFN modules. Extensive experiments on large-scale industrial datasets demonstrate that RankElastor consistently improves recommendation performance, mitigates embedding collapse, and exhibits robust scaling behavior. Code is available at this GitHub repository: https://github.com/vasile-paskardlgm/RankElastor

推荐系统模型扩展嵌入坍缩谱鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。