用凸优化解决强化学习逆问题，更稳定可靠。
Inverse Reinforcement Learning via Convex Optimization
- 将逆强化学习转化为凸优化问题，提升求解稳定性。
- 支持专家轨迹不完美时的鲁棒估计，兼容真实数据。
- 提供自动超参数选择与易用接口，适合非专业用户。
本文研究逆强化学习（IRL）问题，即根据观察到的专家示范估计未知马尔可夫决策过程的奖励函数。现有方法多将IRL建模为非凸优化问题，难以保证鲁棒性和可复现性。我们回顾了Ng和Russel提出的凸形式化方法（CIRL），并重新构建问题，使领域专用语言CVXPY可直接用于指定与求解。同时，针对专家策略以状态-动作轨迹形式给出、可能严重偏离最优的情况，通过增强约束条件扩展了CIRL的应用场景。文中还引入了超参数自动选择的理论分析与实践实现。本说明帮助用户无需掌握凸优化背景即可轻松应用CIRL解决实际问题。
原文摘要 · Abstract (English)
We consider the inverse reinforcement learning (IRL) problem, where an unknown reward function of some Markov decision process is estimated based on observed expert demonstrations. In most existing approaches, IRL is formulated and solved as a nonconvex optimization problem, posing challenges in scenarios where robustness and reproducibility are critical. We discuss a convex formulation of the IRL problem (CIRL) initially proposed by Ng and Russel, and reformulate the problem such that the domain-specific language CVXPY can be applied directly to specify and solve the convex problem. We also extend the CIRL problem to scenarios where the expert policy is not given analytically but by trajectory as state-action pairs, which can be strongly inconsistent with optimality, by augmenting some of the constraints. Theoretical analysis and practical implementation for hyperparameter auto-selection are introduced. This note helps the users to easily apply CIRL for their problems, without background knowledge on convex optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。