用新规则解决私有化强化学习中罕见切换失效问题
When Determinants Are Not Enough: Private Rare Switching
- 改用广义瑞利商设计罕见切换规则，替代传统行列式方法
- 恢复对数级策略更新与常数因子内的置信区间控制
- 适合关注隐私保护强化学习的算法研究者
本文探讨在私有化强化学习场景下，如何将罕见切换机制适配到线性带宽问题中。标准基于行列式的更新规则因设计矩阵单调增长而有效，但引入高斯噪声保障隐私后，单调性被破坏，原有分析不再成立。核心原因是行列式控制体积，而后悔率分析需控制最差方向。为此，提出基于广义瑞利商的新罕见切换规则，重新实现对数级策略更新和常数因子内的置信宽度比较。本文还呈现了该证明的手动整理版本及个人反思。
原文摘要 · Abstract (English)
In this note, I would like to share a small research moment where Codex helped me find the right way to adapt rare switching to the private setting. The standard determinant-based update rule in linear bandits and RL works beautifully because the design matrix grows monotonically. But once Gaussian noise is added for privacy, this monotonicity can fail, and the usual analysis no longer goes through. The key reason is that determinant growth controls volume, while regret analysis needs control of the worst direction. To address this, Codex comes up with a different rare-switching rule based on the generalized Rayleigh quotient, which restores logarithmic policy updates and the desired confidence-width comparison up to a constant factor. I present my manually clean-up version of the proof here as well as some personal reflection on this example.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。