提出通用时域模型，解决离线强化学习中长期规划误差问题。
Offline Reinforcement Learning with Universal Horizon Models
- 用可任意设定时域的模型直接预测未来状态，提升建模灵活性。
- 在100个挑战性任务上优于主流基线，尤其在数据质量差和长程推理任务中表现突出。
- 适合需要长期规划的复杂决策场景，如机器人控制与策略优化。
基于模型的强化学习通过模拟策略轨迹实现价值学习，但在离线强化学习中常因重复模型推断导致误差累积。几何时域模型（GHM）虽能通过直接预测无限时域未来缓解此问题，却难以准确建模远期状态。为此，本文提出通用时域模型（UHM），作为GHM的泛化，可直接对任意时域下的未来状态进行预测。利用该灵活性，我们设计了一种可扩展的价值学习方法，采用截尾时域分布以抑制过大的时域值，稳定训练过程。在100个高难度OGBench任务上的实验表明，该方法显著优于现有基线，尤其在数据严重次优及需长程推理的任务中表现优异。
原文摘要 · Abstract (English)
Model-based reinforcement learning (RL) offers a compelling approach to offline RL by enabling value learning on imagined on-policy trajectories. However, it often suffers from compounding errors due to repeated model inference on self-generated states. While geometric horizon models (GHM) alleviate this issue through direct prediction over a discounted infinite-horizon future, they remain challenged in accurately modeling distant future states. To this end, we introduce universal horizon models (UHM), a generalization of GHM that directly predicts future states under arbitrary horizons. Leveraging this flexibility, we propose a scalable value learning method that employs a winsorized horizon distribution to stabilize training by capping excessively large horizons. Experimental results on 100 challenging OGBench tasks demonstrate that the proposed method outperforms competitive baselines, particularly on tasks with highly suboptimal datasets and those requiring long-horizon reasoning. Project page: https://rllab-snu.github.io/projects/UHM/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。