针对高风险高回报任务,用离散动作+双判别器提升强化学习效果
D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Tasks
- 将连续动作离散化以捕捉多模态行为分布
- 在复杂控制任务中实现比基线更高的成功率和回报
- 适合需要权衡风险与收益的机器人控制场景
高风险高回报(HRHR)任务如障碍跨越常具有多模态动作分布和随机回报。现有强化学习方法通常假设单峰高斯策略并依赖标量值判别器,在此类任务中表现受限。本文正式定义了HRHR任务,并理论证明高斯策略无法保证收敛到最优解。为此提出新框架:(i) 离散化连续动作空间以逼近多模态分布;(ii) 采用熵正则化探索提升对高风险高回报动作的覆盖;(iii) 引入双判别器架构,更准确估计离散动作的价值分布。该框架可扩展至高维动作空间,支持复杂控制任务。在涉及高失败风险的运动与操作基准测试中,本方法优于多个基线,验证了显式建模多模态性与风险的重要性。
原文摘要 · Abstract (English)
Tasks involving high-risk-high-return (HRHR) actions, such as obstacle crossing, often exhibit multimodal action distributions and stochastic returns. Most reinforcement learning (RL) methods assume unimodal Gaussian policies and rely on scalar-valued critics, which limits their effectiveness in HRHR settings. We formally define HRHR tasks and theoretically show that Gaussian policies cannot guarantee convergence to the optimal solution. To address this, we propose a reinforcement learning framework that (i) discretizes continuous action spaces to approximate multimodal distributions, (ii) employs entropy-regularized exploration to improve coverage of risky but rewarding actions, and (iii) introduces a dual-critic architecture for more accurate discrete value distribution estimation. The framework scales to high-dimensional action spaces, supporting complex control domains. Experiments on locomotion and manipulation benchmarks with high risks of failure demonstrate that our method outperforms baselines, underscoring the importance of explicitly modeling multimodality and risk in RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。