对比13个时序预测算法在制造场景的表现,发现简单模型也能胜过复杂模型。
Time-Series Forecasting in Smart Manufacturing Systems: An Experimental Evaluation of the State-of-the-art Algorithms
- 选取13个制造数据集,评估主流时序预测算法在不同场景下的表现。
- MLP类模型在多数任务中表现最佳,长周期预测中PatchTST最稳健。
- 强调模型易用性与资源开销,适合制造业技术人员直接使用。
时序预测(TSF)在制造领域日益重要。尽管已有大量先进算法,但其验证与评估仍不足。本研究在13个制造数据集上评估了当前最先进的时序预测算法,聚焦其在制造场景中的适用性。算法按类别选择以确保代表性,涵盖两类问题和两个预测时长的多种场景。采用WAPE评估性能,并进行事后分析以检验差异显著性。仅使用开源库代码,未做超参数调优,确保模型作为“开箱即用”解决方案的实用性,便于技术背景有限的从业者部署。结果表明,基于Transformer和MLP的架构表现最优,其中MLP类模型在更多场景胜出;单变量任务中PatchTST最为稳健,尤其适用于长周期预测;多变量任务中如N-HITS和TiDE等MLP模型表现更佳。研究还发现,XGBoost等简单模型在特定任务中可超越复杂模型,挑战了模型越复杂效果越好的假设。此外,算法在运行时间和内存消耗方面存在显著差异,凸显计算资源考量的重要性。
原文摘要 · Abstract (English)
TSF is growing in various domains including manufacturing. Although numerous TSF algorithms have been developed recently, the validation and evaluation of algorithms hold substantial value for researchers and practitioners and are missing. This study aims to fill this gap by evaluating the SoTA TSF algorithms on thirteen manufacturing datasets, focusing on their applicability in manufacturing. Each algorithm was selected based on its TSF category to ensure a representative set of algorithms. The evaluation includes different scenarios to evaluate the models using two problem categories and two forecasting horizons. To evaluate the performance, the WAPE was calculated, and additional post hoc analyses were conducted to assess the significance of observed differences. Only algorithms with codes from open-source libraries were utilized, and no hyperparameter tuning was done. This allowed us to evaluate the algorithms as "out-of-the-box" solutions that can be easily implemented, ensuring their usability within the manufacturing by practitioners with limited technical knowledge. This aligns to facilitate the adoption of these techniques in smart manufacturing systems. Based on the results, transformer and MLP-based architectures demonstrated the best performance with MLP-based architecture winning the most scenarios. For univariate TSF, PatchTST emerged as the most robust, particularly for long-term horizons, while for multivariate problems, MLP-based architectures like N-HITS and TiDE showed superior results. The study revealed that simpler algorithms like XGBoost could outperform complex algorithms in certain tasks. These findings challenge the assumption that more sophisticated models produce better results. Additionally, the research highlighted the importance of computational resource considerations, showing variations in runtime and memory usage across different algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。