用分层结构分解复杂任务，让机器人在多实体环境中更高效完成长程目标。
Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion
- 分两层：底层用价值网络选子目标，上层用扩散模型生成具体动作
- 在最难任务上成功率提升超150%，且能适应更长任务和更多实体
- 方法模块化，可兼容现有强化学习算法，适合复杂环境下的智能体训练
我们提出一种分层的以实体为中心的离线目标条件强化学习框架，结合子目标分解与因子化结构，解决多实体环境中长时序任务的挑战。复杂环境中的长程目标达成仍是强化学习的核心难题，尤其在多实体场景下，状态空间具有组合复杂性。目标条件强化学习虽有助于目标泛化与子目标利用，但在高维观测和稀疏奖励下仍面临困难。本工作采用两级架构：基于价值的GCRL代理与因子化子目标生成的条件扩散模型。两者独立训练，通过价值函数选择性生成子目标后组合使用，具备模块化优势，兼容现有GCRL算法。我们在基准任务中引入新变体以凸显多实体场景的挑战，结果表明该方法显著提升底层强化学习代理在图像输入、稀疏奖励下的长程任务表现，在最困难任务上成功率超过150%提升，并可推广至更长时序与更多实体。演示视频见：https://sites.google.com/view/hecrl
原文摘要 · Abstract (English)
We propose a hierarchical entity-centric framework for offline Goal-Conditioned Reinforcement Learning (GCRL) that combines subgoal decomposition with factored structure to solve long-horizon tasks in domains with multiple entities. Achieving long-horizon goals in complex environments remains a core challenge in Reinforcement Learning (RL). Domains with multiple entities are particularly difficult due to their combinatorial complexity. GCRL facilitates generalization across goals and the use of subgoal structure, but struggles with high-dimensional observations and combinatorial state-spaces, especially under sparse reward. We employ a two-level hierarchy composed of a value-based GCRL agent and a factored subgoal-generating conditional diffusion model. The RL agent and subgoal generator are trained independently and composed post hoc through selective subgoal generation based on the value function, making the approach modular and compatible with existing GCRL algorithms. We introduce new variations to benchmark tasks that highlight the challenges of multi-entity domains, and show that our method consistently boosts performance of the underlying RL agent on image-based long-horizon tasks with sparse rewards, achieving over 150% higher success rates on the hardest task in our suite and generalizing to increasing horizons and numbers of entities. Rollout videos are provided at: https://sites.google.com/view/hecrl
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。