用更少参数的MoE模型，在38个表格数据集上超越MLP。
(GG) MoE vs. MLP on Tabular Data
- 引入基于Gumbel-Softmax的门控混合专家模型GG MoE。
- GG MoE在38个数据集上表现最优，且参数量显著更低。
- 适合追求高效、可扩展的表格数据建模者使用。
近年来，大量研究致力于将现代神经网络架构适配到表格数据。然而，尽管参数更多、训练和推理时间更长，这些模型往往无法持续优于基础多层感知机（MLP）。此外，基于MLP的集成模型近期表现出比先进深度学习方法更优的性能与效率。因此，与其构建更深更复杂的模型，我们提出探究是否可用更高效的架构替代MLP而不损失性能。本文首先提出一种基于Gumbel-Softmax门控函数的混合专家模型GG MoE。随后证明，加入嵌入层的GG MoE在38个数据集上的表现优于标准MoE和MLP模型。最后，我们展示MoE及GG MoE所用参数远少于MLP，使其成为可扩展和集成方法的有力候选。
原文摘要 · Abstract (English)
In recent years, significant efforts have been directed toward adapting modern neural network architectures for tabular data. However, despite their larger number of parameters and longer training and inference times, these models often fail to consistently outperform vanilla multilayer perceptron (MLP) neural networks. Moreover, MLP-based ensembles have recently demonstrated superior performance and efficiency compared to advanced deep learning methods. Therefore, rather than focusing on building deeper and more complex deep learning models, we propose investigating whether MLP neural networks can be replaced with more efficient architectures without sacrificing performance. In this paper, we first introduce GG MoE, a mixture-of-experts (MoE) model with a Gumbel-Softmax gating function. We then demonstrate that GG MoE with an embedding layer achieves the highest performance across $38$ datasets compared to standard MoE and MLP models. Finally, we show that both MoE and GG MoE utilize significantly fewer parameters than MLPs, making them a promising alternative for scaling and ensemble methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。