用分布代替动作，统一处理各类控制问题。
Distributions as Actions: A Unified Framework for Diverse Action Spaces
- 将动作建模为参数化分布，使动作空间连续化。
- 新算法在离散、连续和混合任务中表现优异。
- 适合需要统一框架的强化学习研究者。
我们提出一种新型强化学习框架，将参数化动作分布视为动作，重新定义智能体与环境的边界。这一重参数化使新动作空间始终连续，无论原始动作类型是离散、连续或混合。在此框架下，我们开发了广义确定性策略梯度估计器——分布作为动作策略梯度（DA-PG），其方差低于原动作空间的梯度。尽管在分布参数上学习价值函数带来新挑战，我们引入插值评价网络学习（ICL），一种简单有效的策略，其设计受到多臂赌博机理论的启发。基于TD3，我们提出实用的演员-评论家算法——分布作为动作演员-评论家（DA-AC）。实验表明，DA-AC在离散、连续和混合控制等多种场景中均达到有竞争力的性能。
原文摘要 · Abstract (English)
We introduce a novel reinforcement learning (RL) framework that treats parameterized action distributions as actions, redefining the boundary between agent and environment. This reparameterization makes the new action space continuous, regardless of the original action type (discrete, continuous, hybrid, etc.). Under this new parameterization, we develop a generalized deterministic policy gradient estimator, Distributions-as-Actions Policy Gradient (DA-PG), which has lower variance than the gradient in the original action space. Although learning the critic over distribution parameters poses new challenges, we introduce Interpolated Critic Learning (ICL), a simple yet effective strategy to enhance learning, supported by insights from bandit settings. Building on TD3, a strong baseline for continuous control, we propose a practical actor-critic algorithm, Distributions-as-Actions Actor-Critic (DA-AC). Empirically, DA-AC achieves competitive performance in various settings across discrete, continuous, and hybrid control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。