提出首个考虑环境反馈的强化学习优化算法，让策略在自我影响下仍保持最优。
Performative Policy Gradient: Optimality in Performative Reinforcement Learning
- 基于软最大化参数化，设计能应对自反馈环境的策略梯度算法
- 理论证明算法收敛至自适应分布下仍最优的策略
- 在标准测试环境中表现优于现有追求稳定性的算法
部署后的机器学习算法常会改变其作用的环境，从而导致底层动态变化，而传统强化学习方法忽略此现象。尽管监督学习中已研究此类‘行为性’设置下的最优算法设计，但强化学习中的对应问题仍缺乏探索。本文证明了强化学习中行为性版本的性能差分引理与策略梯度定理，并提出首个专为行为性强化学习设计的策略梯度算法——PePG。在软最大化参数化下，无论是否加入熵正则化，均证明PePG可收敛至行为性最优策略，即在自身引发的分布偏移下仍保持最优的策略。该结果显著超越了以往仅实现行为性稳定的算法。此外，在标准行为性强化学习环境上的实验表明，PePG在性能上优于现有追求稳定性的行为性强化学习算法。
原文摘要 · Abstract (English)
Post-deployment machine learning algorithms often influence the environments they act in, and thus shift the underlying dynamics that the standard reinforcement learning (RL) methods ignore. While designing optimal algorithms in this performative setting has recently been studied in supervised learning, the RL counterpart remains under-explored. In this paper, we prove the performative counterparts of the performance difference lemma and the policy gradient theorem in RL, and further introduce the Performative Policy Gradient algorithm (PePG). PePG is the first policy gradient algorithm designed to account for performativity in RL. Under softmax parametrisation, and also with and without entropy regularisation, we prove that PePG converges to performatively optimal policies, i.e. policies that remain optimal under the distribution shifts induced by themselves. Thus, PePG significantly extends the prior works in Performative RL that achieves performative stability but not optimality. Furthermore, our empirical analysis on standard performative RL environments validate that PePG outperforms the existing performative RL algorithms aiming for stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。