分析时间序列归一化对Transformer模型表达能力的影响,发现不同方法效果不一。
Be Wary of Your Time Series Preprocessing
- 提出专用于时间序列的表达能力评估框架,量化模型区分相似与差异输入的能力。
- 理论证明标准差和最小-最大归一化在不同数据下表现差异显著,无最优方案。
- 实验证明有时不归一化反而更好,强调预处理需根据任务定制。
归一化与缩放是时间序列建模中的基础预处理步骤,但在基于Transformer的模型中其理论作用仍不明确。本文首次系统分析了实例级与全局缩放等不同归一化策略对Transformer架构表达能力的影响。我们提出一个专为时间序列设计的表达能力评估框架,量化模型在表示空间中区分相似与不相似输入的能力。基于该框架,推导出标准差归一化和最小-最大归一化两种常用方法的理论边界。分析表明,归一化策略的选择会显著影响模型的表示能力,具体取决于任务和数据特性。我们在多个基于Transformer的模型上,通过分类与预测基准进行了实证验证。结果表明,没有一种归一化方法始终优于其他,某些情况下完全省略归一化反而表现更优。这些发现凸显了预处理在时间序列学习中的关键作用,也推动了针对特定任务和数据集的更合理归一化策略的发展。
原文摘要 · Abstract (English)
Normalization and scaling are fundamental preprocessing steps in time series modeling, yet their role in Transformer-based models remains underexplored from a theoretical perspective. In this work, we present the first formal analysis of how different normalization strategies, specifically instance-based and global scaling, impact the expressivity of Transformer-based architectures for time series representation learning. We propose a novel expressivity framework tailored to time series, which quantifies a model's ability to distinguish between similar and dissimilar inputs in the representation space. Using this framework, we derive theoretical bounds for two widely used normalization methods: Standard and Min-Max scaling. Our analysis reveals that the choice of normalization strategy can significantly influence the model's representational capacity, depending on the task and data characteristics. We complement our theory with empirical validation on classification and forecasting benchmarks using multiple Transformer-based models. Our results show that no single normalization method consistently outperforms others, and in some cases, omitting normalization entirely leads to superior performance. These findings highlight the critical role of preprocessing in time series learning and motivate the need for more principled normalization strategies tailored to specific tasks and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。