用新型门控机制让专家模型更准更快,省去复杂训练
GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts
- 用注意力启发的门控替代传统路由,简化MoE训练
- 无需辅助平衡损失,仍能均衡使用专家且精度更高
- 参数量少、计算效率高,适合真实场景的长短期预测
准确的单变量时间序列预测在能源市场、水文、零售需求和物联网监控等实际系统中至关重要,信号常呈间歇性,预测范围涵盖短长期。尽管变压器和专家混合(MoE)架构日益流行,但关键挑战在于:传统MoE需同时优化主任务损失与辅助负载平衡损失,并精细调节路由温度,阻碍实际应用。本文提出一种新模型架构,简化单变量时间序列预测的训练流程,有效应对长短周期及间歇模式。方法结合稀疏MoE计算与新颖的注意力启发门控机制,取代传统单层Softmax路由。实证表明,该门控设计自然促进专家均衡使用,无需辅助平衡损失即可实现更优预测精度。模型性能优于现有Transformer(如PatchTST),参数量仅为后者的少量。跨多种数据集实验显示,该MoE架构在长短期预测中均比LSTM更高效,支持低成本推理。结果凸显其在高精度与高效计算并重的应用中的潜力。
原文摘要 · Abstract (English)
Accurate univariate forecasting remains a pressing need in real-world systems, such as energy markets, hydrology, retail demand, and IoT monitoring, where signals are often intermittent and horizons span both short- and long-term. While transformers and Mixture-of-Experts (MoE) architectures are increasingly favored for time-series forecasting, a key gap persists: MoE models typically require complicated training with both the main forecasting loss and auxiliary load-balancing losses, along with careful routing/temperature tuning, which hinders practical adoption. In this paper, we propose a model architecture that simplifies the training process for univariate time series forecasting and effectively addresses both long- and short-term horizons, including intermittent patterns. Our approach combines sparse MoE computation with a novel attention-inspired gating mechanism that replaces the traditional one-layer softmax router. Through extensive empirical evaluation, we demonstrate that our gating design naturally promotes balanced expert utilization and achieves superior predictive accuracy without requiring the auxiliary load-balancing losses typically used in classical MoE implementations. The model achieves better performance while utilizing only a fraction of the parameters required by state-of-the-art transformer models, such as PatchTST. Furthermore, experiments across diverse datasets confirm that our MoE architecture with the proposed gating mechanism is more computationally efficient than LSTM for both long- and short-term forecasting, enabling cost-effective inference. These results highlight the potential of our approach for practical time-series forecasting applications where both accuracy and computational efficiency are critical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。