通过截断回报提升CVaR策略梯度的样本效率
Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation
- 用回报截断替代丢弃轨迹，更高效利用数据
- 在多个环境中性能优于基线方法
- 适合追求鲁棒性与数据效率的强化学习研究者
使用策略梯度优化条件风险价值（CVaR）时，现有方法需丢弃大量轨迹，导致样本效率低下。本文提出一种新公式：对训练所用轨迹的总回报进行截断，而非直接丢弃。我们证明，当截断值设置合适时，该方法等价于原问题。在多个环境中的实验证明，该方法相比基线表现更优且稳定。代码已公开于 https://github.com/HarryMJMead/cvar-return-capping。
原文摘要 · Abstract (English)
When optimising for conditional value at risk (CVaR) using policy gradients (PG), current methods rely on discarding a large proportion of trajectories, resulting in poor sample efficiency. We propose a reformulation of the CVaR optimisation problem by capping the total return of trajectories used in training, rather than simply discarding them, and show that this is equivalent to the original problem if the cap is set appropriately. We show, with empirical results in an number of environments, that this reformulation of the problem results in consistently improved performance compared to baselines. We have made all our code available here: https://github.com/HarryMJMead/cvar-return-capping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。