黑盒环境下操纵强化学习训练数据，诱导模型学坏策略。
Online Poisoning Attack Against Reinforcement Learning under Black-box Environments
- 通过随机梯度法近似未知环境梯度，实现对奖励与状态转移的污染。
- 构建约束优化模型并用惩罚法转化，避免双重采样问题。
- 适用于攻击者不知环境细节、目标算法灵活的现实场景。
本文提出一种针对黑盒环境下强化学习代理的在线环境投毒算法，攻击者通过故意篡改训练数据，引导智能体学习错误策略。与以往主要关注白盒设置的研究不同，本工作聚焦于攻击者未知环境动态、目标代理采用灵活强化学习算法的场景。提出一种可同时污染奖励函数和状态转移的攻击方案，将投毒任务建模为带约束的优化问题，沿用文献[ma2019policy]框架。由于攻击者无法获知转移概率，采用基于样本估计的随机梯度下降法逼近真实梯度。结合惩罚法与双层重构，将原问题转化为无约束形式，有效规避双重采样难题。在迷宫环境中验证了该算法的有效性。
原文摘要 · Abstract (English)
This paper proposes an online environment poisoning algorithm tailored for reinforcement learning agents operating in a black-box setting, where an adversary deliberately manipulates training data to lead the agent toward a mischievous policy. In contrast to prior studies that primarily investigate white-box settings, we focus on a scenario characterized by \textit{unknown} environment dynamics to the attacker and a \textit{flexible} reinforcement learning algorithm employed by the targeted agent. We first propose an attack scheme that is capable of poisoning the reward functions and state transitions. The poisoning task is formalized as a constrained optimization problem, following the framework of \cite{ma2019policy}. Given the transition probabilities are unknown to the attacker in a black-box environment, we apply a stochastic gradient descent algorithm, where the exact gradients are approximated using sample-based estimates. A penalty-based method along with a bilevel reformulation is then employed to transform the problem into an unconstrained counterpart and to circumvent the double-sampling issue. The algorithm's effectiveness is validated through a maze environment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。