提出抗奖励污染的鲁棒Q-learning算法，保障在恶意干扰下仍能稳定学习。
Robust Q-Learning under Corrupted Rewards
- 利用历史奖励数据构建鲁棒的贝尔曼算子，抵抗部分观测奖励被篡改
- 在对抗污染比例ε下，收敛误差仅增加可忽略的O(ε)项
- 适用于奖励分布无界但二阶矩有界的复杂环境，适合高可靠性场景
近期对无模型强化学习算法的非渐近行为分析日益增多，但其在非理想环境（如存在污染奖励）下的表现仍不清楚。本文研究了经典Q-learning算法在强污染攻击模型下的鲁棒性，其中对手可任意扰动少量观测奖励。我们证明，原始Q-learning可能产生任意大的误差。为此，提出一种新型鲁棒同步Q-learning算法，通过历史奖励数据构造每步的鲁棒经验贝尔曼算子。理论证明该算法具有有限时间收敛速率，在无攻击情况下达到现有最优边界，仅增加与对抗污染比例ε相关的微小不可避免误差O(ε)。值得注意的是，即使真实奖励分布支持集无限，只要二阶矩有界，结论依然成立。
原文摘要 · Abstract (English)
Recently, there has been a surge of interest in analyzing the non-asymptotic behavior of model-free reinforcement learning algorithms. However, the performance of such algorithms in non-ideal environments, such as in the presence of corrupted rewards, is poorly understood. Motivated by this gap, we investigate the robustness of the celebrated Q-learning algorithm to a strong-contamination attack model, where an adversary can arbitrarily perturb a small fraction of the observed rewards. We start by proving that such an attack can cause the vanilla Q-learning algorithm to incur arbitrarily large errors. We then develop a novel robust synchronous Q-learning algorithm that uses historical reward data to construct robust empirical Bellman operators at each time step. Finally, we prove a finite-time convergence rate for our algorithm that matches known state-of-the-art bounds (in the absence of attacks) up to a small inevitable $O(\varepsilon)$ error term that scales with the adversarial corruption fraction $\varepsilon$. Notably, our results continue to hold even when the true reward distributions have infinite support, provided they admit bounded second moments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。