提出方差缩减策略，提升多目标强化学习采样效率。
Variance Reduced Policy Gradient Method for Multi-Objective Reinforcement Learning
- 引入方差缩减技术优化多目标策略梯度。
- 在不强假设下显著降低样本需求量。
- 适合需平衡多个目标的复杂决策场景。
多目标强化学习（MORL）是传统强化学习的推广，旨在同时优化多个常冲突的目标，而非单一奖励。该方法在需权衡多种目标的复杂决策场景中至关重要，如在提升性能的同时控制成本。本文研究使用非线性标量化函数组合目标的MORL问题。与标准强化学习类似，策略梯度方法（PGMs）在处理大规模连续状态-动作空间时表现优异。然而，现有MORL的PGM存在高样本低效问题，需大量数据才能有效。此前改进方法依赖过强假设，削弱了PGM在大规模空间中的可扩展性优势。本文通过引入方差缩减技术，在保持一般性假设的前提下，降低策略梯度的样本复杂度，显著提升采样效率。
原文摘要 · Abstract (English)
Multi-Objective Reinforcement Learning (MORL) is a generalization of traditional Reinforcement Learning (RL) that aims to optimize multiple, often conflicting objectives simultaneously rather than focusing on a single reward. This approach is crucial in complex decision-making scenarios where agents must balance trade-offs between various goals, such as maximizing performance while minimizing costs. We consider the problem of MORL where the objectives are combined using a non-linear scalarization function. Just like in standard RL, policy gradient methods (PGMs) are amongst the most effective for handling large and continuous state-action spaces in MORL. However, existing PGMs for MORL suffer from high sample inefficiency, requiring large amounts of data to be effective. Previous attempts to solve this problem rely on overly strict assumptions, losing PGMs' benefits in scalability to large state-action spaces. In this work, we address the issue of sample efficiency by implementing variance-reduction techniques to reduce the sample complexity of policy gradients while maintaining general assumptions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。