用逆优化思想解决离线强化学习，提升数据效率与稳定性
Offline Reinforcement Learning via Inverse Optimization
- 引入凸的次优损失函数，基于逆优化设计新算法
- 在MuJoCo上可靠恢复教师策略，参数量少一个数量级
- 适合样本受限场景，代码开源可复现
受逆优化在多个领域成功应用的启发，我们提出一种针对连续状态和动作空间的新型离线强化学习算法，利用逆优化文献中的凸损失函数——次优损失。为缓解离线强化学习中常见的分布偏移问题，我们采用一种鲁棒且非因果的模型预测控制（MPC）专家，通过事后信息修正动态模型的名义模型。与现有方法不同，该鲁棒MPC专家具备精确且可计算的凸重构形式。研究第二部分表明，由所提凸损失训练的逆优化假设类具有充分表达能力，在MuJoCo基准测试中可靠恢复教师行为。尽管参数量仅为基线方法的数个数量级，该方法在样本受限条件下仍表现竞争力。为促进结果复现，我们提供了开源实现及实验代码，项目地址：https://github.com/TolgaOk/offlineRLviaIO。
原文摘要 · Abstract (English)
Inspired by the recent successes of Inverse Optimization (IO) across various application domains, we propose a novel offline Reinforcement Learning (ORL) algorithm for continuous state and action spaces, leveraging the convex loss function called ``sub-optimality loss'' from the IO literature. To mitigate the distribution shift commonly observed in ORL problems, we further employ a robust and non-causal Model Predictive Control (MPC) expert steering a nominal model of the dynamics using in-hindsight information stemming from the model mismatch. Unlike the existing literature, our robust MPC expert enjoys an exact and tractable convex reformulation. In the second part of this study, we show that the IO hypothesis class, trained by the proposed convex loss function, enjoys ample expressiveness and {reliably recovers teacher behavior in MuJoCo benchmarks. The method achieves competitive results compared to widely-used baselines in sample-constrained settings, despite using} orders of magnitude fewer parameters. To facilitate the reproducibility of our results, we provide an open-source package implementing the proposed algorithms and the experiments. The code is available at https://github.com/TolgaOk/offlineRLviaIO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。