针对复杂任务中策略梯度易陷入局部最优的问题，提出新方法提升决策质量。
Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions
- 多候选动作选择+截断低效局部最优，引导梯度上升
- 在复杂控制任务中找到最优动作频率更高
- 适合高维动作空间或存在约束的强化学习场景
在强化学习中，如DDPG和TD3等离线策略演员-批评方法使用确定性策略梯度：从环境数据中学习Q函数，演员通过梯度上升最大化该函数。我们观察到，在复杂任务（如灵巧操作和受约束移动）中，Q函数存在大量局部最优，导致梯度上升容易陷入其中。为此，我们提出SAVO演员架构，(i) 生成多个动作提案并选择Q值最高的一个；(ii) 通过反复截断劣质局部最优来近似Q函数，更有效地引导梯度上升。我们在受限移动、灵巧操作及大离散动作空间推荐系统等任务上进行了评估，结果表明，我们的演员能更频繁地找到最优动作，性能优于其他演员架构。
原文摘要 · Abstract (English)
In reinforcement learning, off-policy actor-critic methods like DDPG and TD3 use deterministic policy gradients: the Q-function is learned from environment data, while the actor maximizes it via gradient ascent. We observe that in complex tasks such as dexterous manipulation and restricted locomotion with mobility constraints, the Q-function exhibits many local optima, making gradient ascent prone to getting stuck. To address this, we introduce SAVO, an actor architecture that (i) generates multiple action proposals and selects the one with the highest Q-value, and (ii) approximates the Q-function repeatedly by truncating poor local optima to guide gradient ascent more effectively. We evaluate tasks such as restricted locomotion, dexterous manipulation, and large discrete-action space recommender systems and show that our actor finds optimal actions more frequently and outperforms alternate actor architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。