分层强化学习结合状态抽象，提升复杂网格环境探索效率。
Decoupled Hierarchical Reinforcement Learning with State Abstraction for Discrete Grids
- 分层双级结构：高层用RL决策，低层用规则策略，解耦探索与执行。
- 在两个自定义网格环境中，探索效率和累积奖励均优于PPO。
- 适合大规模离散状态空间的强化学习任务，尤其关注探索难题。
在复杂离散状态空间环境下，有效探索仍是强化学习的核心挑战，尤其是在部分可观测场景中。本文提出一种解耦分层强化学习框架（DcHRL-SA），融合状态抽象技术以应对该问题。该方法采用双层级架构：高层为基于RL的策略代理，低层为基于规则的策略，促进高效探索；同时引入状态抽象方法对离散状态进行聚类，显著降低状态维度。在两个定制化的离散网格环境中进行实验，结果表明该方法在探索效率、收敛速度、累积奖励和策略稳定性方面均持续优于PPO。该研究展示了在大规模探索空间的离散网格中，解耦分层策略与状态抽象相结合的可行性。代码将公开于 https://github.com/XQY169/DcHRL-SA。
原文摘要 · Abstract (English)
Effective agent exploration remains a core challenge in reinforcement learning (RL) for complex discrete state-space environments, particularly under partial observability. This paper presents a decoupled hierarchical RL framework integrating state abstraction (DcHRL-SA) to address this issue. The proposed method employs a dual-level architecture, consisting of a high level RL-based actor and a low-level rule-based policy, to promote effective exploration. Additionally, state abstraction method is incorporated to cluster discrete states, effectively lowering state dimensionality. Experiments conducted in two discrete customized grid environments demonstrate that the proposed approach consistently outperforms PPO in terms of exploration efficiency, convergence speed, cumulative reward, and policy stability. These results demonstrate a practical approach for integrating decoupled hierarchical policies and state abstraction in discrete grids with large-scale exploration space. Code will be available at https://github.com/XQY169/DcHRL-SA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。