KL正则化可让强化学习和博弈决策自动满足差分隐私，无需额外加噪。
KL-regularization Itself is Differentially Private in Bandits and RLHF
- 用KL正则化改进目标函数，使策略采样动作自然具备差分隐私
- 在多臂老虎机、线性上下文老虎机和人类反馈强化学习中均有效
- 既保隐私又不损失性能，适合关注隐私保护的强化学习应用
差分隐私（DP）为数据驱动算法提供严格隐私保障，确保输出在仅一项数据不同的数据集间难以区分。通常需显式注入噪声以实现DP，但现有算法的内在随机性提供了‘免费’实现隐私的可能性。本文研究正则化在三种决策问题中的隐私作用：多臂老虎机、线性上下文老虎机与离线设置下的从人类反馈中强化学习（RLHF）。结果表明，在学习目标中加入KL正则化（优化中的常见做法），可使生成的随机策略所采动作本身满足差分隐私。该方法无需额外噪声注入，同时保留了正则化提升性能的优势，为隐私保障提供了新路径。
原文摘要 · Abstract (English)
Differential Privacy (DP) provides a rigorous framework for privacy, ensuring the outputs of data-driven algorithms remain statistically indistinguishable across datasets that differ in a single entry. While guaranteeing DP generally requires explicitly injecting noise either to the algorithm itself or to its outputs, the intrinsic randomness of existing algorithms presents an opportunity to achieve DP ``for free''. In this work, we explore the role of regularization in achieving DP across three different decision-making problems: multi-armed bandits, linear contextual bandits, and reinforcement learning from human feedback (RLHF), in offline data settings. We show that adding KL-regularization to the learning objective (a common approach in optimization algorithms) makes the action sampled from the resulting stochastic policy itself differentially private. This offers a new route to privacy guarantees without additional noise injection, while also preserving the inherent advantage of regularization in enhancing performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。