现有时间序列评估标准误导了模型进步,真实性能需更复杂数据验证。
Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains
- 用多样化非平稳数据替代单一周期性数据做评测
- 简单统计模型在常见数据上常不输深度学习模型
- 适合关注模型真实性能的研究者与工业界应用者
当前时间序列预测模型的评估主要依赖于具有强而持久周期性与季节性的基准数据集,这掩盖了高效经典方法的真实表现。我们发现这些‘标准’数据集通常呈现显著自相关与季节循环,可被线性或统计模型有效捕捉,导致复杂深度学习架构在此类数据上往往不优于经典方法,引发其计算开销与模型复杂度是否值得的疑问。我们呼吁社区(1)淘汰或大幅扩充现有基准,引入包含结构突变、时变波动、概念漂移等非平稳特征,以及来自多元真实场景中更具挑战性的动态数据;(2)要求每个深度学习提交结果必须包含针对下游任务特性的稳健经典与简单基线模型。唯有如此,才能确保报告的性能提升反映真实的科学方法进步,而非仅因评测基准偏向重复模式学习能力所致。
原文摘要 · Abstract (English)
We argue that the current practice of evaluating AI/ML time-series forecasting models, predominantly on benchmarks characterized by strong, persistent periodicities and seasonalities, obscures real progress by overlooking the performance of efficient classical methods. We demonstrate that these "standard" datasets often exhibit dominant autocorrelation patterns and seasonal cycles that can be effectively captured by simpler linear or statistical models, rendering complex deep learning architectures frequently no more performant than their classical counterparts for these specific data characteristics, and raising questions as to whether any marginal improvements justify the significant increase in computational overhead and model complexity. We call on the community to (I) retire or substantially augment current benchmarks with datasets exhibiting a wider spectrum of non-stationarities, such as structural breaks, time-varying volatility, and concept drift, and less predictable dynamics drawn from diverse real-world domains, and (II) require every deep learning submission to include robust classical and simple baselines, appropriately chosen for the specific characteristics of the downstream tasks' time series. By doing so, we will help ensure that reported gains reflect genuine scientific methodological advances rather than artifacts of benchmark selection favoring models adept at learning repetitive patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。