用Cramér距离改进强化学习，让模型更稳更准
Distributional Reinforcement Learning via the Cramér Distance
- 用分布式方法建模动作价值，最小化Cramér距离优化
- 在复杂机器人任务中超越SAC和现有分布方法
- 高方差目标触发保守更新，抑制过估计问题
本文将软演员-评论家（Soft Actor-Critic, SAC）算法拓展至分布式强化学习框架，提出一种基于Cramér距离的分布式软演员-评论家算法（C-DSAC）。该方法通过分布形式表示状态-动作值，并以平方Cramér距离作为优化目标。在多个机器人基准测试中，所提算法性能优于基线SAC及当前主流分布式方法，尤其在高复杂度环境下优势更为显著。分析表明，其优越性部分源于“置信度驱动”的Q值更新机制：当目标分布方差较大（置信度低）时，模型更新更加保守，有效缓解了值函数过估计问题。本工作深化了对分布式强化学习的理解，揭示了影响收敛与估值精度的算法机制。
原文摘要 · Abstract (English)
This paper explores the application of the Soft Actor-Critic (SAC) algorithm within a Distributional Reinforcement Learning setting and introduces an implementation of such algorithm named Cramér-based Distributional Soft Actor-Critic (C-DSAC). The novel approach employs distributional reinforcement learning to represent state-action values, and minimizes the squared Cramér distance for learning the distribution. Empirical results across various robotic benchmarks indicate that our algorithm surpasses the performance of baseline SAC and contemporary distributional methods, with the performance advantage becoming increasingly pronounced in high-complexity environments. To explain the efficiency of the new approach, we conduct an analysis showing that its superior performance is partly due to \textit{confidence-driven} Q-value updates: High-variance target distributions (low confidence in target) lead to more conservative model updates, thereby attenuating the impact of overestimated values. This work deepens the understanding of distributional reinforcement learning, offering insights into the algorithmic mechanisms governing convergence and value estimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。