用同态加密实现安全高效的强化学习，解决云端隐私计算难题。
Efficient Implementation of Reinforcement Learning over Homomorphic Encryption
- 采用无比较结构的正则化强化学习，适配同态加密计算
- 在网格环境上验证加密Z-learning收敛，误差可控
- 适合需要云外包且保护数据隐私的智能控制场景
本文研究基于云端的加密控制策略合成。尽管已有加密控制实现，但本文聚焦较少探索的隐私保护型控制合成，其计算量大，适合云外包。将控制策略合成分为基于模型、仿真驱动和数据驱动三类，分析其在全同态加密（FHE）下的实现。关键挑战在于强化学习中常见的最小/最大比较操作难以在加密数据上执行。为此，本文关注相对熵正则化强化学习，因其无需比较，更易在FHE下实现。我们展示了线性可解值迭代、路径积分控制与Z-learning均可直接在FHE上实现。通过在网格世界环境中使用CKKS加密方案进行数值模拟，验证了加密Z-learning的收敛性与可接受的近似误差。结果表明，该方法具有实现安全高效云端强化学习的潜力。
原文摘要 · Abstract (English)
We investigate encrypted control policy synthesis over the cloud. While encrypted control implementations have been studied previously, we focus on the less explored paradigm of privacy-preserving control synthesis, which can involve heavier computations ideal for cloud outsourcing. We classify control policy synthesis into model-based, simulator-driven, and data-driven approaches and examine their implementation over fully homomorphic encryption (FHE) for privacy enhancements. A key challenge arises from comparison operations (min or max) in standard reinforcement learning algorithms, which are difficult to execute over encrypted data. This observation motivates our focus on Relative-Entropy-regularized reinforcement learning (RL) problems, which simplifies encrypted evaluation of synthesis algorithms due to their comparison-free structures. We demonstrate how linearly solvable value iteration, path integral control, and Z-learning can be readily implemented over FHE. We conduct a case study of our approach through numerical simulations of encrypted Z-learning in a grid world environment using the CKKS encryption scheme, showing convergence with acceptable approximation error. Our work suggests the potential for secure and efficient cloud-based reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。