用离散分类方式表示动作，让强化学习更稳定高效
RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning
- 将动作维度转为离散分桶分布，类似分类任务优化
- 在多个连续控制任务中显著超越传统高斯策略性能
- 适合追求稳定训练和高精度控制的RL研究者
持续控制中的在线强化学习仍是主流范式，但标准实现依赖高斯策略和浅层MLP，常因梯度噪声导致优化不稳定，政策更新需保守。本文将策略表示视为首要设计选择，提出离散分类策略：将每个动作维度建模为离散区间的概率分布，诱导出类似分类交叉熵的目标函数。结合监督学习的架构进展，进一步引入正则化网络，构建RN-D模型。在多个连续控制基准测试中，仅替换标准高斯策略即显著提升性能，达成在线强化学习当前最优结果。代码已开源：https://github.com/alwaysbyx/RND-RL。
原文摘要 · Abstract (English)
On-policy Reinforcement Learning (RL) remains a dominant paradigm for continuous control, yet standard implementations rely on Gaussian actors and relatively shallow MLP policies, often leading to brittle optimization when gradients are noisy, and policy updates must be conservative. In this paper, we revisit actor policy representation as a first-class design choice for on-policy RL. We study discretized categorical actors, which represent each action dimension as a distribution over discrete bins and induce a policy objective analogous to classification cross-entropy loss. Building on architectural advances from supervised learning, we further pair discretized categorical actors with regularized networks, yielding RN-D. Across diverse continuous-control benchmarks, we show that simply replacing the standard Gaussian actor with our proposed actor substantially improves performance, achieving state-of-the-art results within on-policy RL. We release our code at https://github.com/alwaysbyx/RND-RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。