arXiv:2505.08768cs.LG2025-05被引 1

通过敏感度评估剪枝注意力模块,显著降低时序模型计算开销。

SPAT: Sensitivity-based Multihead-attention Pruning on Time Series Forecasting Models

  • 基于动态敏感度指标SEND,精准识别冗余注意力头并整体移除。
  • 剪枝后模型在多变量数据集上减少35.274%算力消耗,误差下降超1.9%。
  • 适合追求高效推理的工业部署,尤其对资源受限场景友好。

基于注意力的架构在多变量时间序列预测中表现优异,但计算成本高昂。现有方法如分块和自适应掩码虽能缩减模型规模与延迟,但仍存在局限。本文提出结构化剪枝方法SPAT(Sensitivity-based Pruner for Attention),通过在预训练阶段使用动态敏感度度量SEND,识别并移除冗余的注意力模块,实现高效率模型。与以往仅剪枝部分参数的方法不同,SPAT直接删除整个注意力模块,降低过拟合风险,并无需专用硬件即可提速。在多变量数据集上的实验表明,SPAT剪枝模型在均方误差(MSE)上降低2.842%,平均绝对误差(MAE)降低1.996%,浮点运算次数(FLOPs)减少35.274%。此外,其性能优于当前主流轻量级、Mamba及大语言模型基线方法,在标准与零样本推理任务中均表现更优,凸显保留关键注意力机制的重要性。代码已公开:https://anonymous.4open.science/r/SPAT-6042。

原文摘要 · Abstract (English)

Attention-based architectures have achieved superior performance in multivariate time series forecasting but are computationally expensive. Techniques such as patching and adaptive masking have been developed to reduce their sizes and latencies. In this work, we propose a structured pruning method, SPAT ($\textbf{S}$ensitivity $\textbf{P}$runer for $\textbf{At}$tention), which selectively removes redundant attention mechanisms and yields highly effective models. Different from previous approaches, SPAT aims to remove the entire attention module, which reduces the risk of overfitting and enables speed-up without demanding specialized hardware. We propose a dynamic sensitivity metric, $\textbf{S}$ensitivity $\textbf{E}$nhanced $\textbf{N}$ormalized $\textbf{D}$ispersion (SEND) that measures the importance of each attention module during the pre-training phase. Experiments on multivariate datasets demonstrate that SPAT-pruned models achieve reductions of 2.842% in MSE, 1.996% in MAE, and 35.274% in FLOPs. Furthermore, SPAT-pruned models outperform existing lightweight, Mamba-based and LLM-based SOTA methods in both standard and zero-shot inference, highlighting the importance of retaining only the most effective attention mechanisms. We have made our code publicly available https://anonymous.4open.science/r/SPAT-6042.

注意力剪枝时序预测模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。