提出BraVE方法，高效解决离线强化学习中的高维离散动作空间难题。
BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces
- 用树状结构遍历动作空间，仅评估线性数量的联合动作
- 在超四百万动作环境中性能提升最高达20倍
- 适合需要高效处理复杂动作依赖的离线强化学习场景
高维离散动作空间下的离线强化学习因联合动作空间随子动作数量呈指数增长，且子动作间依赖关系复杂而极具挑战。现有方法或需穷举所有动作（计算不可行），或因子化Q值无法表达联合子动作影响。我们提出分支价值估计（BraVE），一种基于价值的方法，通过树状结构动作遍历，仅评估线性数量的联合动作，同时保留子动作依赖结构。BraVE在含超过四百万动作的环境中，性能相较先前离线RL方法最高提升20倍。
原文摘要 · Abstract (English)
Offline reinforcement learning in high-dimensional, discrete action spaces is challenging due to the exponential scaling of the joint action space with the number of sub-actions and the complexity of modeling sub-action dependencies. Existing methods either exhaustively evaluate the action space, making them computationally infeasible, or factorize Q-values, failing to represent joint sub-action effects. We propose Branch Value Estimation (BraVE), a value-based method that uses tree-structured action traversal to evaluate a linear number of joint actions while preserving dependency structure. BraVE outperforms prior offline RL methods by up to $20\times$ in environments with over four million actions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。