arXiv:2607.24025cs.IRcs.LG2026-07

针对推荐系统中嵌入与注意力崩溃问题,提出谱感知Transformer提升模型性能与可扩展性。

SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation

论文配图:SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation
图 1 · 摘自论文原文
  • 通过可学习谱平滑模块动态调整嵌入特征分布
  • 在均匀谱分布空间中建模特征交互,提升注意力有效秩
  • 适合深度推荐系统,尤其对长尾数据有显著优化

Transformer在多个领域表现卓越,但直接将其自注意力机制用于推荐时常表现不佳,甚至落后于简单模型。本文揭示该瓶颈源于推荐场景特有的嵌入与注意力崩溃,其根源是数据异质性和长尾分布导致的严重谱崩溃,即少数主奇异值主导。理论证明此现象会引发前向与反向传播中的恶性循环,加速崩溃并限制模型深度扩展能力。为此,提出SpecFormer——一种面向推荐的谱感知Transformer:1)引入可学习谱平滑模块,动态平滑输入嵌入的奇异值分布;2)设计谱平滑注意力机制,在更均匀的谱空间中建模特征交互;3)基于奇异值泰勒展开的谱残差位置编码,显式提供谱归纳偏置。在工业级及两个公开数据集上的实验表明,SpecFormer显著优于现有基线。值得注意的是,该模型已成功部署于真实商业推荐系统,且堆叠更多层能持续提升注意力有效秩与推荐性能。

原文摘要 · Abstract (English)

Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models. In this paper, we reveal that this performance bottleneck stems from severe embedding and attention collapse unique to recommendation scenarios. The heterogeneity and long-tail nature of recommendation data lead to a severe spectral collapse dominated by a few principal singular values. We further theoretically demonstrate that this triggers a vicious cycle in recommendation model's forward and backward propagation, which accelerates embedding and attention collapse and limits the model's scaling capability with increased depth. To address these issues, we propose SpecFormer, a novel Spectral-Aware Transformer designed for mitigating embedding and attention collapse in recommendation. Specifically, SpecFormer introduces 1) a Learnable Spectral Softening module to dynamically smooth the singular values distribution of the input token embeddings; 2) a Spectrum-softened Attention mechanism to model feature interaction under a more uniform spectral distribution space; 3) a Spectral Residual Position Encoding via Taylor expansion of singular values, explicitly providing a spectral inductive bias for feature interactions. Extensive experiments on one industrial and two public datasets demonstrate that SpecFormer significantly outperforms state-of-the-art baselines. Notably, SpecFormer has been successfully deployed in a real-world commercial recommender system and exhibits exceptional scaling capabilities: stacking SpecFormer layers actively improves the attention effective rank and recommendation performance.

推荐系统Transformer谱分析注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。