多步预测可缓解模型偏差导致的误差累积，提升控制性能。
Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Error
- 用多步预测直接学习系统动态，避免单步模型递推误差积累。
- 在模型不完全可观时，多步预测显著降低偏差，优于单步模型。
- 适合模型存在部分可观测性问题的强化学习与模仿学习场景。
误差累积是基于学习的控制中的主要挑战，常限制模型无关强化学习和模仿学习的性能。一种常见方法是直接训练多步预测器，而非依赖单步模型的自回归滚动。然而，多步预测的优势何时超过其模型复杂度的代价尚不明确。本文针对线性动态系统，严格分析了这一权衡：当模型类正确且准确捕捉系统动态时，单步模型具有更低的渐近预测误差；而在模型因部分可观测性而误设时，直接多步预测器能显著减少偏差，表现优于单步方法。数值实验验证了理论结果，还评估了一种中间策略——使用多步损失训练单步模型，以及在闭环控制设置中单步与多步预测器的性能表现。
原文摘要 · Abstract (English)
Compounding error, where small prediction mistakes accumulate over time, presents a major challenge in learning-based control. For example, this issue often limits the performance of model-based reinforcement learning and imitation learning. One common approach to mitigate compounding error is to train multi-step predictors directly, rather than relying on autoregressive rollout of a single-step model. However, it is not well understood when the benefits of multi-step prediction outweigh the added complexity of learning a more complicated model. In this work, we provide a rigorous analysis of this trade-off in the context of linear dynamical systems. We show that when the model class is well-specified and accurately captures the system dynamics, single-step models achieve lower asymptotic prediction error. On the other hand, when the model class is misspecified due to partial observability, direct multi-step predictors can significantly reduce bias and thus outperform single-step approaches. These theoretical results are supported by numerical experiments, wherein we also (a) empirically evaluate an intermediate strategy which trains a single-step model using a multi-step loss and (b) evaluate performance of single step and multi-step predictors in a closed loop control setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。