揭示时间序列大模型在分布内外的缩放规律,指导模型设计与扩展。
Towards Neural Scaling Laws for Time Series Foundation Models
- 对比编码器和解码器架构的缩放特性,分析参数、算力与数据量影响。
- 发现分布内与分布外的损失缩放趋势相似,但架构显著影响性能提升幅度。
- 为构建更强大时间序列基础模型提供可落地的缩放设计指南。
缩放定律为时间序列基础模型(TSFMs)的设计提供了重要参考。然而,以往研究主要关注模型在分布内(ID)数据上的缩放规律,对其分布外(OOD)行为及模型架构的影响仍缺乏深入探索。本文研究了两种常见架构——仅编码器与仅解码器的Transformer,并在不同参数量、计算预算和数据规模下评估其在ID与OOD数据上的表现。实验表明,TSFM的对数似然损失在OOD与ID设置下具有相似的缩放行为。进一步对比不同架构的缩放特性,以两种先进TSFM为例,发现编码器架构比解码器架构更具可扩展性;而两个先进模型的架构改进虽提升了分布内性能,却削弱了分布外可扩展性。本文通过整合发现,填补了对TSFM缩放规律理解的空白,为设计和扩展具备更强能力的大规模时间序列模型提供了实用指导。
原文摘要 · Abstract (English)
Scaling laws offer valuable insights into the design of time series foundation models (TSFMs). However, previous research has largely focused on the scaling laws of TSFMs for in-distribution (ID) data, leaving their out-of-distribution (OOD) scaling behavior and the influence of model architectures less explored. In this work, we examine two common TSFM architectures, encoder-only and decoder-only Transformers, and investigate their scaling behavior on both ID and OOD data. These models are trained and evaluated across varying parameter counts, compute budgets, and dataset sizes. Our experiments reveal that the log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID settings. We further compare the scaling properties across different architectures, incorporating two state-of-the-art TSFMs as case studies, showing that model architecture plays a significant role in scaling. The encoder-only Transformers demonstrate better scalability than the decoder-only Transformers, while the architectural enhancements in the two advanced TSFMs primarily improve ID performance but reduce OOD scalability. While scaling up TSFMs is expected to drive performance breakthroughs, the lack of a comprehensive understanding of TSFM scaling laws has hindered the development of a robust framework to guide model scaling. We fill this gap in this work by synthesizing our findings and providing practical guidelines for designing and scaling larger TSFMs with enhanced model capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。