arXiv:2511.07237cs.LGcs.CL2025-11被引 2

发现时间序列模型越大规模反而越差,提出只留关键层就能提速提效。

The Few Govern the Many:Unveiling Few-Layer Dominance for Time Series Models

  • 找出模型中仅少数层起作用,其余冗余甚至干扰训练。
  • 保留21%参数可提升12%准确率,推理快2.7倍。
  • 适配8个主流模型,95%任务表现更好或相当。

大规模模型在时间序列预测中占据主导地位,主要依赖两类范式:微调基于文本的大语言模型(LLM4TS)和从零训练时间序列基础模型(TSFM)。两者均假设模型规模与数据量越大,性能越好。然而我们观察到时间序列模型存在显著的「缩放悖论」——模型越大,性能反而越差。通过在两个模型家族、四种规模(100M至1.7B参数)及多种数据(最多60亿观测)上进行大量实验,我们证实该现象普遍存在。进一步分析内部表征,发现一种称为「少层主导」的现象:仅有少量层具备实际功能,其余多数层冗余、未被充分利用,甚至干扰训练。据此提出一种自动识别并保留主导层的方法。在我们的模型中,仅保留21%参数即可实现最高12%的精度提升和2.7倍的推理加速。在8个主流先进模型(包括LLM4TS与TSFM,参数量90M至60亿)上验证了方法普适性,保留少于30%层在超过95%的任务中达到或超越原模型性能。

原文摘要 · Abstract (English)

Large-scale models are at the forefront of time series (TS) forecasting, dominated by two paradigms: fine-tuning text-based Large Language Models (LLM4TS) and training Time Series Foundation Models (TSFMs) from scratch. Both approaches share a foundational assumption that scaling up model capacity and data volume leads to improved performance. However, we observe a \textit{\textbf{scaling paradox}} in TS models, revealing a puzzling phenomenon that larger models do \emph{NOT} achieve better performance. Through extensive experiments on two model families across four scales (100M to 1.7B parameters) and diverse data (up to 6B observations), we rigorously confirm that the scaling paradox is a pervasive issue. We then diagnose its root cause by analyzing internal representations, identifying a phenomenon we call \textit{few-layer dominance}: only a small subset of layers are functionally important, while the majority are redundant, under-utilized, and can even distract training. Based on this discovery, we propose a practical method to automatically identify and retain only these dominant layers. In our models, retaining only 21\% of the parameters achieves up to a 12\% accuracy improvement and a 2.7$\times$ inference speedup. We validate the universality of our method on 8 prominent SOTA models (LLM4TS and TSFMs, 90M to 6B), showing that retaining less than 30\% of layers achieves comparable or superior accuracy in over 95\% of tasks.

时间序列模型压缩少层主导缩放悖论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。