机器学习预测的误差与系统动态行为有关,高误差常出现在复杂且不稳定的系统状态中。
Dynamical errors in machine learning forecasts
- 用瞬时维度和反持续性两个动态指标分析预测误差来源
- 高误差对应更高复杂度和更低稳定性,长期预测偏差更明显
- 提出基于动态指标的新误差评估方法,适合科学建模与可靠性分析
在机器学习预测中,均方误差(MSE)和平均绝对误差(MAE)等标准误差指标仅衡量预测值与真实值的偏差,但无法直接评估预测结果在物理或动力学层面的一致性。本文研究了这些传统误差指标与系统内在动力学特性之间的关系,采用瞬时维度(d)和反持续性(θ)两个近期发展的动力学指标。结果表明,更高的预测误差(如更高MSE)通常出现在瞬时维度更大(更高复杂度)和反持续性更高(更低持久性)的状态中。为进一步评估动力学一致性,我们提出了基于这些动力学指标的误差度量方法,用于衡量预测的d和θ与真实值之间的偏差。利用该方法,我们在三个经典数据集(Lorenz、Kuramoto-Sivashinsky方程、Kolmogorov流)及一个真实气象预测任务上分析了直接预测与递归预测策略。结果显示,机器学习预测在长预报时效或长时间递归模拟下存在显著的动力学性质失真,为模型可信度评估提供了补充信息,有助于改进模型性能。
原文摘要 · Abstract (English)
In machine learning forecasting, standard error metrics such as mean absolute error (MAE) and mean squared error (MSE) quantify discrepancies between predictions and target values. However, these metrics do not directly evaluate the physical and/or dynamical consistency of forecasts, an increasingly critical concern in scientific and engineering applications. Indeed, a fundamental yet often overlooked question is whether machine learning forecasts preserve the dynamical behavior of the underlying system. Addressing this issue is essential for assessing the fidelity of machine learning models and identifying potential failure modes, particularly in applications where maintaining correct dynamical behavior is crucial. In this work, we investigate the relationship between standard forecasting error metrics, such as MAE and MSE, and the dynamical properties of the underlying system. To achieve this goal, we use two recently developed dynamical indices: the instantaneous dimension ($d$), and the inverse persistence ($θ$). Our results indicate that larger forecast errors -- e.g., higher MSE -- tend to occur in states with higher $d$ (higher complexity) and higher $θ$ (lower persistence). To further assess dynamical consistency, we propose error metrics based on the dynamical indices that measure the discrepancy of the forecasted $d$ and $θ$ versus their correct values. Leveraging these dynamical indices-based metrics, we analyze direct and recursive forecasting strategies for three canonical datasets -- Lorenz, Kuramoto-Sivashinsky equation, and Kolmogorov flow -- as well as a real-world weather forecasting task. Our findings reveal substantial distortions in dynamical properties in ML forecasts, especially for long forecast lead times or long recursive simulations, providing complementary information on ML forecast fidelity that can be used to improve ML models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。