arXiv:2502.06037cs.LG2025-02被引 6

探究时间序列大模型能否组合推理,而非仅记忆模式。

Investigating Compositional Reasoning in Time Series Foundation Models

  • 定义并测试时间序列模型的组合推理能力,区分于泛化能力。
  • 基于补丁的Transformer表现最佳,计算量仅为MLP模型的3%。
  • 部分模型在零样本外分布场景下超越传统统计基线。

大规模预训练时间序列基础模型(TSFMs)在多个领域展现出出色的零样本性能。然而,其成功是源于对训练数据中模式的记忆,还是具备对这些模式进行推理的能力?尽管推理在大语言模型研究中备受关注,但在时间序列模型中尚无明确定义且几乎未被探索。受语言建模文献启发,本文正式定义了预测中的组合推理,并将其与分布内泛化区分开来。我们在多个合成与真实世界数据集上评估了16种主流深度学习预测模型的推理与泛化能力。通过受控实验,系统分析了7个开源TSFM中设计选择对推理能力的影响。研究发现,基于补丁的Transformer在推理方面表现最优,紧随其后的是残差化MLP架构,其浮点运算量(FLOPs)仅为前者97%更低,可训练参数量减少86%。有趣的是,在某些零样本外分布场景中,这些模型的表现优于在分布内数据上训练的移动平均和指数平滑等统计基线。仅有少数设计选择(如标记化方法)对Transformer模型性能产生显著负面影响。

原文摘要 · Abstract (English)

Large pre-trained time series foundation models (TSFMs) have demonstrated promising zero-shot performance across a wide range of domains. However, a question remains: Do TSFMs succeed by memorizing patterns in training data, or do they possess the ability to reason about such patterns? While reasoning is a topic of great interest in the study of Large Language Models (LLMs), it is undefined and largely unexplored in the context of TSFMs. In this work, inspired by language modeling literature, we formally define compositional reasoning in forecasting and distinguish it from in-distribution generalization. We evaluate the reasoning and generalization capabilities of 16 popular deep learning forecasting models on multiple synthetic and real-world datasets. Additionally, through controlled studies, we systematically examine which design choices in 7 popular open-source TSFMs contribute to improved reasoning capabilities. Our study yields key insights into the impact of TSFM architecture design on compositional reasoning and generalization. We find that patch-based Transformers have the best reasoning performance, closely followed by residualized MLP-based architectures, which are 97\% less computationally complex in terms of FLOPs and 86\% smaller in terms of the number of trainable parameters. Interestingly, in some zero-shot out-of-distribution scenarios, these models can outperform moving average and exponential smoothing statistical baselines trained on in-distribution data. Only a few design choices, such as the tokenization method, had a significant (negative) impact on Transformer model performance.

时间序列组合推理Transformer基础模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。