新算法让强化学习在超大离散动作空间中更高效稳定。
Breaking the Grid: Distance-Guided Reinforcement Learning in Large Discrete Action Spaces
- 用距离引导探索和更新，把优化转为稳定回归任务。
- 在最多10^20个动作的空间中，性能提升最高达66%。
- 适合物流、调度等复杂决策场景，也支持连续+离散混合动作。
强化学习在物流、排程和推荐系统等大规模决策问题中应用日益广泛，但传统算法在超大离散动作空间下面临维数灾难。本文提出距离引导强化学习（DGRL），结合采样动态邻域与基于距离的更新机制，实现高达10^20个动作空间中的高效强化学习。不同于以往方法，DGRL通过随机体积探索，将策略优化转化为稳定的回归任务，使梯度方差与动作空间规模解耦。在结构化任务中，DGRL可保证局部价值提升。该方法天然适用于混合连续-离散动作空间。实验表明，在规则与非规则结构环境中，其性能相较最先进基准最高提升66%，同时加快收敛速度并降低计算复杂度。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) is increasingly applied to large-scale decision-making problems like logistics, scheduling, and recommender systems, but existing algorithms struggle with the curse of dimensionality in such large discrete action spaces. We propose Distance-Guided Reinforcement Learning (DGRL), combining Sampled Dynamic Neighborhoods and Distance-Based Updates to enable efficient RL in problems with up to $10^{20}$ actions. Unlike prior methods, DGRL performs stochastic volumetric exploration and transforms policy optimization into a stable regression task, decoupling gradient variance from action space cardinality. On structured tasks, DGRL provably guarantees local value improvement. DGRL naturally generalizes to hybrid continuous-discrete action spaces. We demonstrate performance improvements of up to 66% against state-of-the-art benchmarks across regularly and irregularly structured environments, while simultaneously improving convergence speed and computational complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。