arXiv:2410.14630cs.LGcs.AI2024-10中稿 · TMLR被引 2

为时间序列预测中的可学习嵌入设计正则化方法,提升模型泛化能力。

On the Regularization of Learnable Embeddings for Time Series Forecasting

  • 通过嵌入扰动和周期重置等正则化策略,防止局部参数过拟合
  • 在主流架构上验证,正则化使预测性能持续提升
  • 适合研究时间序列基础模型与泛化能力的学者参考

在多时间序列预测中,捕捉每条序列的独特特征颇具挑战。现代深度学习方法通常结合共享(全局)模型与针对每条序列的局部层,后者常以可学习嵌入实现。理想情况下,这些局部嵌入应编码序列特有的动态信息。然而,若它们作为端到端模型的参数被直接学习,可能仅充当序列标识符,导致共享模块过度依赖此类标识,降低其在新场景下的可迁移性。本文首次对局部可学习嵌入的正则化进行系统性实证研究,表明此类正则化能稳定提升广泛采用架构的性能。特别地,通过嵌入扰动来防止局部与全局参数共适应的方法尤为有效,我们还测试了训练过程中周期性重置嵌入的策略。结果揭示了可学习局部参数与共享处理层之间的关键互动机制,为构建高效的时间序列基础模型提供了重要洞见。

原文摘要 · Abstract (English)

In forecasting multiple time series, accounting for the individual features of each sequence can be challenging. To address this, modern deep learning methods for time series analysis combine a shared (global) model with local layers, specific to each time series, often implemented as learnable embeddings. Ideally, these local embeddings should encode meaningful representations of the unique dynamics of each sequence. However, when these are learned end-to-end as parameters of a forecasting model, they may end up acting as mere sequence identifiers. Shared processing blocks may then become reliant on such identifiers, limiting their transferability to new contexts. In this paper, we address this issue by investigating methods to regularize the learning of local learnable embeddings for time series processing. Specifically, we perform the first extensive empirical study on the subject and show how such regularizations consistently improve performance in widely adopted architectures. Furthermore, we show that methods attempting to prevent the co-adaptation of local and global parameters by means of embeddings perturbation are particularly effective in this context. In this regard, we include in the comparison several perturbation-based regularization methods, going as far as periodically resetting the embeddings during training. The obtained results provide an important contribution to understanding the interplay between learnable local parameters and shared processing layers: a key challenge in modern time series processing models and a step toward developing effective foundation models for time series.

时间序列嵌入正则模型泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。