对比剪枝方法在Transformer时间序列预测中的效果，发现高稀疏度剪枝可提升性能。
A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting
- 对比无结构与结构化剪枝在多变量时间序列模型上的表现
- 部分模型在高达高稀疏度下剪枝后性能优于原始模型
- 剪枝后需微调，且结构化剪枝对推理加速作用有限
当前时间序列预测领域以Transformer模型为主导。其高参数量和计算资源需求给低功耗嵌入式设备的实际部署带来挑战。剪枝是降低神经网络参数量和节省计算量的成熟方法。然而，剪枝对基于Transformer的时间序列预测模型的影响和优势尚不明确。为填补这一空白，我们通过评估多种前沿多变量时间序列模型上的无结构与结构化剪枝，开展了一项比较性基准研究。重点考察剪枝策略对模型预测性能及模型大小、计算量、推理时间等计算指标的影响。结果表明，某些模型可在极高稀疏度下进行剪枝，且剪枝后性能优于原密集模型；但剪枝后必须进行微调。此外，即使具备相应软硬件支持，结构化剪枝也无法带来显著的推理时间节省。
原文摘要 · Abstract (English)
The current landscape in time-series forecasting is dominated by Transformer-based models. Their high parameter count and corresponding demand in computational resources pose a challenge to real-world deployment, especially for commercial and scientific applications with low-power embedded devices. Pruning is an established approach to reduce neural network parameter count and save compute. However, the implications and benefits of pruning Transformer-based models for time series forecasting are largely unknown. To close this gap, we provide a comparative benchmark study by evaluating unstructured and structured pruning on various state-of-the-art multivariate time series models. We study the effects of these pruning strategies on model predictive performance and computational aspects like model size, operations, and inference time. Our results show that certain models can be pruned even up to high sparsity levels, outperforming their dense counterpart. However, fine-tuning pruned models is necessary. Furthermore, we demonstrate that even with corresponding hardware and software support, structured pruning is unable to provide significant time savings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。