揭示DNN在特征交互模型中防维度坍塌的作用机制
Understanding DNNs in Feature Interaction Models: A Dimensional Collapse Perspective
- 从维度鲁棒性视角分析DNN如何缓解嵌入维度坍塌
- 实验证明并行与堆叠DNN均能有效抑制维度坍塌
- 提供梯度理论分析，解释为何DNN提升表示稳定性
深度神经网络（DNN）在特征交互推荐模型中广泛应用，但其作用仍存争议。一方面，有研究认为DNN能隐式捕捉高阶特征交互；另一方面，近期工作指出DNN在学习点积（尤其是二阶交互）方面存在局限，更难建模高阶交互。本文提出新视角：从维度鲁棒性理解DNN的有效性。通过大量实验评估并行与堆叠DNN在两类特征交互模型上的表现，并进行组件级消融分析。结果表明，两类DNN均能有效缓解嵌入的维度坍塌。结合梯度理论分析与实证证据，揭示了维度坍塌的内在机制。
原文摘要 · Abstract (English)
DNNs have gained widespread adoption in feature interaction recommendation models. However, there has been a longstanding debate on their roles. On one hand, some works claim that DNNs possess the ability to implicitly capture high-order feature interactions. Conversely, recent studies have highlighted the limitations of DNNs in effectively learning dot products, specifically second-order interactions, let alone higher-order interactions. In this paper, we present a novel perspective to understand the effectiveness of DNNs: their impact on the dimensional robustness of the representations. In particular, we conduct extensive experiments involving both parallel DNNs and stacked DNNs. Our evaluation encompasses an overall study of complete DNN on two feature interaction models, alongside a fine-grained ablation analysis of components within DNNs. Experimental results demonstrate that both parallel and stacked DNNs can effectively mitigate the dimensional collapse of embeddings. Furthermore, a gradient-based theoretical analysis, supported by empirical evidence, uncovers the underlying mechanisms of dimensional collapse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。