通过大规模并行与参数调控，让策略值函数高效优化复杂连续控制任务。
Massively Scaling Explicit Policy-conditioned Value Functions
- 用大批次、权重裁剪和扰动缩放实现策略空间的高效探索
- 在自定义Ant环境中性能媲美PPO和SAC等先进算法
- 首次将动作级参数表示与专用网络结构引入DRL策略值函数
我们提出一种显式策略条件价值函数（EPVF）的大规模扩展策略，显著提升其在复杂连续控制任务中的表现。EPVF学习一个显式依赖于策略参数的值函数V(θ)，支持对任意策略参数进行梯度更新。但其大规模应用面临参数无限制增长和策略空间探索效率低的问题。为此，我们采用基于GPU的并行仿真器、大批次训练、权重裁剪和缩放扰动。结果表明，该方法可成功解决如自定义Ant环境等复杂任务，并在性能上达到Proximal Policy Optimization（PPO）和Soft Actor-Critic（SAC）等前沿深度强化学习基线水平。我们还探索了此前工作中的基于动作的策略参数表示及专用神经网络架构，首次将其应用于DRL中的策略空间特征处理。
原文摘要 · Abstract (English)
We introduce a scaling strategy for Explicit Policy-Conditioned Value Functions (EPVFs) that significantly improves performance on challenging continuous-control tasks. EPVFs learn a value function V(θ) that is explicitly conditioned on the policy parameters, enabling direct gradient-based updates to the parameters of any policy. However, EPVFs at scale struggle with unrestricted parameter growth and efficient exploration in the policy parameter space. To address these issues, we utilize massive parallelization with GPU-based simulators, big batch sizes, weight clipping and scaled peturbations. Our results show that EPVFs can be scaled to solve complex tasks, such as a custom Ant environment, and can compete with state-of-the-art Deep Reinforcement Learning (DRL) baselines like Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC). We further explore action-based policy parameter representations from previous work and specialized neural network architectures to efficiently handle weight-space features, which have not been used in the context of DRL before.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。