揭示对抗逆强化学习中奖励迁移的条件,提升高维环境下的迁移效果。
On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theory
- 基于随机矩阵理论分析转移矩阵特征分布,推导奖励可迁移的充要条件。
- 证明在高维状态空间下,该条件以高概率成立,即使转移矩阵不可观测。
- 提出混合策略框架,结合策略优化与软演员-评论家算法,显著提升迁移性能。
在单个专家的逆强化学习(IRL)背景下,对抗逆强化学习(AIRL)作为基础方法,旨在提供全面且可迁移的任务描述。然而,其实际性能受限于框架的理想化分解条件、奖励恢复均衡的模糊证明,或在高维环境中的鲁棒性不足。本文研究了状态空间趋于无穷的高维场景。首先,通过分析转移矩阵减去单位矩阵后的秩,建立奖励可迁移的必要充分条件;进而利用随机矩阵理论,分析该矩阵的谱分布,证明该秩准则在转移矩阵不可观测时仍以高概率成立。这表明迁移限制并非来自AIRL框架本身,而是源于其内部强化学习算法的训练方差。基于此,提出一种混合框架:在源环境中使用基于策略的近端策略优化,在目标环境中采用基于策略的软演员-评论家算法,显著提升了奖励迁移的有效性。
原文摘要 · Abstract (English)
In the context of inverse reinforcement learning (IRL) with a single expert, adversarial inverse reinforcement learning (AIRL) serves as a foundational approach to providing comprehensive and transferable task descriptions. However, AIRL faces practical performance challenges, primarily stemming from the framework's overly idealized decomposability condition, the unclear proof regarding the potential equilibrium in reward recovery, or questionable robustness in high-dimensional environments. This paper revisits AIRL in \textbf{high-dimensional scenarios where the state space tends to infinity}. Specifically, we first establish a necessary and sufficient condition for reward transferability by examining the rank of the matrix derived from subtracting the identity matrix from the transition matrix. Furthermore, leveraging random matrix theory, we analyze the spectral distribution of this matrix, demonstrating that our rank criterion holds with high probability even when the transition matrices are unobservable. This suggests that the limitations on transfer are not inherent to the AIRL framework itself, but are instead related to the training variance of the reinforcement learning algorithms employed within it. Based on this insight, we propose a hybrid framework that integrates on-policy proximal policy optimization in the source environment with off-policy soft actor-critic in the target environment, leading to significant improvements in reward transfer effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。