在线快速推断对手目标,适用于实时对抗场景
Recursive Deep Inverse Reinforcement Learning
- 用二阶牛顿法递归更新,类似扩展卡尔曼滤波
- 在基准任务中准确恢复专家策略的代价函数
- 适合需要实时响应的攻防系统、战略游戏等
从对手行为中推断其目标,在网络安全、军事和策略游戏中对反规划至关重要。基于最大熵原理的深度逆强化学习方法虽有潜力,但通常为离线模式,需大批量数据和梯度下降,依赖一阶更新,难以用于实时场景。本文提出一种在线递归深度逆强化学习(RDIRL)方法,通过递归最小化引导代价学习(GCL)目标的上界,采用序列二阶牛顿更新,类似扩展卡尔曼滤波(EKF),实现快速收敛。实验表明,RDIRL可在标准与对抗性基准任务中准确恢复专家代理的代价与奖励函数,性能优于多个主流IRL算法。
原文摘要 · Abstract (English)
Inferring an adversary's goals from exhibited behavior is crucial for counterplanning and non-cooperative multi-agent systems in domains like cybersecurity, military, and strategy games. Deep Inverse Reinforcement Learning (IRL) methods based on maximum entropy principles show promise in recovering adversaries goals but are typically offline, require large batch sizes with gradient descent, and rely on first-order updates, limiting their applicability in real-time scenarios. We propose an online Recursive Deep Inverse Reinforcement Learning (RDIRL) approach to recover the cost function governing the adversary actions and goals. Specifically, we minimize an upper bound on the standard Guided Cost Learning (GCL) objective using sequential second-order Newton updates, akin to the Extended Kalman Filter (EKF), leading to a fast (in terms of convergence) learning algorithm. We demonstrate that RDIRL is able to recover cost and reward functions of expert agents in standard and adversarial benchmark tasks. Experiments on benchmark tasks show that our proposed approach outperforms several leading IRL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。