用密集层替代序列混洗器,发现简单结构也能高效处理时间序列
JustDense: Just using Dense instead of Sequence Mixer for Time Series analysis
- 用密集层替换原模型中的序列混洗器,简化架构设计
- 在29个基准上测试,性能与复杂模型相当甚至更优
- 挑战深度复杂模型必然更好的认知,适合模型轻量化研究者
序列混洗器和通道混洗器作为序列模型的核心机制,已成为时间序列分析(TSA)的主流。然而,近期研究质疑复杂序列混洗器(如注意力机制)的必要性,表明简单架构也可达到相当或更优性能,暗示复杂混洗器的优势可能源于其他架构或优化因素。基于此,我们提出核心问题:时间序列分析是否必须使用常见序列混洗器?为此,我们提出JustDense,一项系统性实验,将多个知名TSA模型中的序列混洗器替换为密集层。基于MatrixMixer框架,JustDense将序列混洗器视为混合矩阵,并以密集层替代,从而隔离混洗操作,建立清晰的理论基础。我们在7个先进TSA模型上,针对5类代表性任务,在29个基准上进行广泛实验。结果表明,用密集层替代序列混洗器可获得相当甚至更优性能。当专用序列混洗器仍具优势时,JustDense挑战了‘更深更复杂架构天然更好’的假设。
原文摘要 · Abstract (English)
Sequence and channel mixers, the core mechanism in sequence models, have become the de facto standard in time series analysis (TSA). However, recent studies have questioned the necessity of complex sequence mixers, such as attention mechanisms, demonstrating that simpler architectures can achieve comparable or even superior performance. This suggests that the benefits attributed to complex sequencemixers might instead emerge from other architectural or optimization factors. Based on this observation, we pose a central question: Are common sequence mixers necessary for time-series analysis? Therefore, we propose JustDense, an empirical study that systematically replaces sequence mixers in various well-established TSA models with dense layers. Grounded in the MatrixMixer framework, JustDense treats any sequence mixer as a mixing matrix and replaces it with a dense layer. This substitution isolates the mixing operation, enabling a clear theoretical foundation for understanding its role. Therefore, we conducted extensive experiments on 29 benchmarks covering five representative TSA tasks using seven state-of-the-art TSA models to address our research question. The results show that replacing sequence mixers with dense layers yields comparable or even superior performance. In the cases where dedicated sequence mixers still offer benefits, JustDense challenges the assumption that "deeper and more complex architectures are inherently better" in TSA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。