通过结构化剪枝让时间序列大模型更专注，性能超越专用小模型。
Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning
- 先剪枝再微调，聚焦有效参数子结构。
- 在6个基准上，剪枝后微调性能显著提升，部分达最优。
- 适合想用大模型但资源有限的研究者或工程师。
扩展规律推动了时间序列基础模型（TSFMs）的发展，这些模型通过大规模预训练，在零样本预测中表现优异。然而，即使经过微调，TSFMs 仍无法稳定超越在全量下游数据上训练的小型专用模型。关键问题是如何有效适配 TSFMs 到特定预测任务。通过对多种 TSFMs 的实证研究发现，预训练模型存在计算上的内在稀疏性和冗余，表明其已学会激活与任务相关的网络子结构以应对多样化预测任务。为保留这一宝贵先验知识，我们提出一种结构化剪枝方法，通过限制后续微调过程在更相关且紧凑的参数空间内进行。在七个 TSFMs 和六个基准上的大量实验表明，对较小的剪枝后模型进行微调，其预测性能显著优于原始模型。该‘剪枝-微调’范式常使 TSFMs 达到当前最佳表现，并超越强大的专用基线。源代码公开于 https://github.com/SJTU-DMTai/Prune-then-Finetune。
原文摘要 · Abstract (English)
Scaling laws motivate the development of Time Series Foundation Models (TSFMs) that pre-train vast parameters and achieve remarkable zero-shot forecasting performance. Surprisingly, even after fine-tuning, TSFMs cannot consistently outperform smaller, specialized models trained on full-shot downstream data. A key question is how to realize effective adaptation of TSFMs for a target forecasting task. Through empirical studies on various TSFMs, the pre-trained models often exhibit inherent sparsity and redundancy in computation, suggesting that TSFMs have learned to activate task-relevant network substructures to accommodate diverse forecasting tasks. To preserve this valuable prior knowledge, we propose a structured pruning method to regularize the subsequent fine-tuning process by focusing it on a more relevant and compact parameter space. Extensive experiments on seven TSFMs and six benchmarks demonstrate that fine-tuning a smaller, pruned TSFM significantly improves forecasting performance compared to fine-tuning original models. This prune-then-finetune paradigm often enables TSFMs to achieve state-of-the-art performance and surpass strong specialized baselines. Source code is made publicly available at https://github.com/SJTU-DMTai/Prune-then-Finetune.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。