提出首个满足洗牌隐私的强化学习算法,兼顾隐私与性能。
Near-Optimal Reinforcement Learning with Shuffle Differential Privacy
- 采用洗牌隐私模型,无需信任中心服务器。
- 实现近似最优误差,性能接近集中式模型。
- 适合需要隐私保护的网络化决策系统。
强化学习(RL)是序列决策的强大工具,但其交互数据常引发隐私担忧,尤其在复杂网络系统中,从操作和用户数据中学习可能暴露系统信息。现有差分隐私(DP)模型存在局限:集中式模型依赖完全可信服务器,存在单点故障风险;本地模型则导致显著性能下降,不适用于多数网络应用。本文通过新兴的洗牌隐私模型——一种介于两者之间的中间信任模型——解决这一问题,该模型提供强隐私保障且无需中心信任假设。我们提出首个基于策略消除的洗牌差分隐私强化学习算法(SDP-PE),引入新颖的指数级批处理调度与“遗忘”机制,平衡隐私与学习性能。理论分析表明,SDP-PE达到近似最优后悔界,实现优于本地模型、媲美集中式模型的隐私-后悔权衡。数值实验验证了理论结果,证明了算法有效性。本工作确立了洗牌模型在网络安全数据驱动决策中的可行性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is a powerful tool for sequential decision-making, but its application is often hindered by privacy concerns arising from its interaction data. This challenge is particularly acute in advanced networked systems, where learning from operational and user data can expose systems to privacy inference attacks. Existing differential privacy (DP) models for RL are often inadequate: the centralized model requires a fully trusted server, creating a single point of failure risk, while the local model incurs significant performance degradation that is unsuitable for many networked applications. This paper addresses this gap by leveraging the emerging shuffle model of privacy, an intermediate trust model that provides strong privacy guarantees without a centralized trust assumption. We present Shuffle Differentially Private Policy Elimination (SDP-PE), the first generic policy elimination-based algorithm for episodic RL under the shuffle model. Our method introduces a novel exponential batching schedule and a ``forgetting'' mechanism to balance the competing demands of privacy and learning performance. Our analysis shows that SDP-PE achieves a near-optimal regret bound, demonstrating a superior privacy-regret trade-off with utility comparable to the centralized model while significantly outperforming the local model. The numerical experiments also corroborate our theoretical results and demonstrate the effectiveness of SDP-PE. This work establishes the viability of the shuffle model for secure data-driven decision-making in networked systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。