用知识蒸馏让轻量MLP学会复杂模型的长期预测能力
TimeDistill: Efficient Long-Term Time Series Forecasting with MLP via Cross-Architecture Distillation
- 用跨架构蒸馏，让MLP从Transformer/CNN学多尺度时频模式
- 性能提升最高18.6%，推理速度提升7倍，参数量减少130倍
- 适合需要高效部署的长时序预测场景
基于Transformer和CNN的方法在长期时间序列预测中表现优异，但其高计算与存储开销限制了大规模部署。为解决此问题，我们提出将轻量级MLP与先进架构结合，通过知识蒸馏（KD）实现知识迁移。初步研究表明，不同模型可捕捉互补的时频域模式，如多尺度与多周期特征。基于此，我们提出TimeDistill——一种跨架构知识蒸馏框架，将教师模型（如Transformer、CNN）中的这些模式迁移到MLP。此外，我们提供了理论分析，表明该KD方法可视为一种特殊形式的mixup数据增强。TimeDistill使MLP性能提升最高达18.6%，在八个数据集上超越教师模型；推理速度最快提升7倍，参数量减少130倍。我们还进行了广泛评估，验证了TimeDistill的通用性与有效性。
原文摘要 · Abstract (English)
Transformer-based and CNN-based methods demonstrate strong performance in long-term time series forecasting. However, their high computational and storage requirements can hinder large-scale deployment. To address this limitation, we propose integrating lightweight MLP with advanced architectures using knowledge distillation (KD). Our preliminary study reveals different models can capture complementary patterns, particularly multi-scale and multi-period patterns in the temporal and frequency domains. Based on this observation, we introduce TimeDistill, a cross-architecture KD framework that transfers these patterns from teacher models (e.g., Transformers, CNNs) to MLP. Additionally, we provide a theoretical analysis, demonstrating that our KD approach can be interpreted as a specialized form of mixup data augmentation. TimeDistill improves MLP performance by up to 18.6%, surpassing teacher models on eight datasets. It also achieves up to 7X faster inference and requires 130X fewer parameters. Furthermore, we conduct extensive evaluations to highlight the versatility and effectiveness of TimeDistill.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。