用博弈论方法解决多目标强化学习中的最差情况优化问题
Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach
- 将多目标问题转为双人零和博弈，用镜像下降法优化策略
- 理论证明算法收敛，且在离散与深度强化学习中表现更优
- 适合需要平衡多个目标的复杂决策场景
本文提出一种可证明收敛且实用的多目标强化学习框架，采用最大最小准则。从博弈论视角出发，将最大最小多目标强化学习重构为两人零和正则化连续博弈，并设计基于镜像下降的高效算法。该方法简化了策略更新过程，同时保证全局逐次迭代收敛。我们提供了全面的理论分析，涵盖精确与近似策略评估下的迭代复杂度，以及样本复杂度边界。为进一步提升性能，引入自适应正则化机制。实验表明，在表格型设置中算法表现出良好收敛性；在深度强化学习实现中，于多个多目标强化学习环境显著优于先前基线。
原文摘要 · Abstract (English)
In this paper, we propose a provably convergent and practical framework for multi-objective reinforcement learning with max-min criterion. From a game-theoretic perspective, we reformulate max-min multi-objective reinforcement learning as a two-player zero-sum regularized continuous game and introduce an efficient algorithm based on mirror descent. Our approach simplifies the policy update while ensuring global last-iterate convergence. We provide a comprehensive theoretical analysis on our algorithm, including iteration complexity under both exact and approximate policy evaluations, as well as sample complexity bounds. To further enhance performance, we modify the proposed algorithm with adaptive regularization. Our experiments demonstrate the convergence behavior of the proposed algorithm in tabular settings, and our implementation for deep reinforcement learning significantly outperforms previous baselines in many MORL environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。