轻量级时间序列预测模型,用弱信息提升精度并加速推理。
Towards Lightweight Time Series Forecasting: a Patch-wise Transformer with Weak Data Enriching
- 用轻量注意力机制替代复杂模块,减少参数和计算开销。
- 在9个数据集上精度超越现有方法,参数量减少且推理速度提升至1/3。
- 可无缝集成弱监督信息,适合边缘设备部署与通用模型增强。
基于分块Transformer的时间序列预测虽精度优异,但依赖复杂结构与海量参数,导致训练与推理成本高昂,难以在资源受限的边缘设备上部署。同时,现有方法多为自回归式,仅使用历史值,忽略易获取的上下文信息(如天气、日期、时段)。为此,提出LiPFormer:一种轻量级分块Transformer,结合弱数据增强。首先,采用新型轻量跨块注意力与线性变换注意力,移除层归一化与前馈网络等高耗组件;其次,设计轻量弱数据增强模块,利用无需人工标注的外部上下文信息提供弱监督信号,不显著增加复杂度,支持即插即用。在9个基准数据集上的实验表明,LiPFormer在精度上优于现有方法,参数量、训练时长与GPU内存占用均显著降低。边缘设备部署显示,其推理时间仅为经典Transformer的1/3。此外,弱数据增强可通用集成于多种Transformer模型中,有效提升精度,展现良好泛化性。
原文摘要 · Abstract (English)
Patch-wise Transformer based time series forecasting achieves superior accuracy. However, this superiority relies heavily on intricate model design with massive parameters, rendering both training and inference expensive, thus preventing their deployments on edge devices with limited resources and low latency requirements. In addition, existing methods often work in an autoregressive manner, which take into account only historical values, but ignore valuable, easy-to-obtain context information, such as weather forecasts, date and time of day. To contend with the two limitations, we propose LiPFormer, a novel Lightweight Patch-wise Transformer with weak data enriching. First, to simplify the Transformer backbone, LiPFormer employs a novel lightweight cross-patch attention and a linear transformation-based attention to eliminate Layer Normalization and Feed Forward Network, two heavy components in existing Transformers. Second, we propose a lightweight, weak data enriching module to provide additional, valuable weak supervision to the training. It enhances forecasting accuracy without significantly increasing model complexity as it does not involve expensive, human-labeling but using easily accessible context information. This facilitates the weak data enriching to plug-and-play on existing models. Extensive experiments on nine benchmark time series datasets demonstrate that LiPFormer outperforms state-of-the-art methods in accuracy, while significantly reducing parameter scale, training duration, and GPU memory usage. Deployment on an edge device reveals that LiPFormer takes only 1/3 inference time compared to classic Transformers. In addition, we demonstrate that the weak data enriching can integrate seamlessly into various Transformer based models to enhance their accuracy, suggesting its generality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。