针对可分解动作空间的离线强化学习，提出新方法并构建首个公开基准数据集。
An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces
- 基于DecQN的价值分解框架，设计适配可分解动作空间的离线算法
- 在自建多质量、多复杂度数据集上验证方法有效性，提升价值估计准确性
- 开源全部数据与代码，推动可复现研究，适合关注离线强化学习的开发者
将强化学习拓展至离线场景具有广阔前景，尤其在数据采集困难或高风险领域。离线强化学习成功的关键在于缓解状态-动作对在数据中缺失时的价值估计过高偏差。尽管近年已有多种方法提出，但多集中于连续或小规模离散动作空间，而现实中许多问题天然具有可分解的动作结构，该方向却未受足够关注。本文首次系统研究可分解动作空间下的离线强化学习，以DecQN中的价值分解为基础，论证分解式方法的合理性，并在自定义的多质量、多复杂度数据集上，全面评估数种适配该场景的离线技术。为支持可复现研究，所有数据集与代码均已开源。
原文摘要 · Abstract (English)
Expanding reinforcement learning (RL) to offline domains generates promising prospects, particularly in sectors where data collection poses substantial challenges or risks. Pivotal to the success of transferring RL offline is mitigating overestimation bias in value estimates for state-action pairs absent from data. Whilst numerous approaches have been proposed in recent years, these tend to focus primarily on continuous or small-scale discrete action spaces. Factorised discrete action spaces, on the other hand, have received relatively little attention, despite many real-world problems naturally having factorisable actions. In this work, we undertake a formative investigation into offline reinforcement learning in factorisable action spaces. Using value-decomposition as formulated in DecQN as a foundation, we present the case for a factorised approach and conduct an extensive empirical evaluation of several offline techniques adapted to the factorised setting. In the absence of established benchmarks, we introduce a suite of our own comprising datasets of varying quality and task complexity. Advocating for reproducible research and innovation, we make all datasets available for public use alongside our code base.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。