解决随机控制中梯度估计发散问题,实现稳定学习的样本效率突破。
Sample Complexity of Policy Gradient for Log-Growth Control
- 利用奇对称性配对观测,消除梯度估计中的发散项。
- 在已知噪声密度时达到理论最优的 𝑂̃(1/η) 样本复杂度。
- 适用于高阶光滑噪声分布,适合鲁棒控制与强化学习研究者。
我们研究了对数增长控制下的策略梯度样本复杂度——即从观测状态转移中学习反馈增益,以最优稳定受乘性噪声驱动的标量线性系统。目标函数𝐽(𝐾)=𝔼[log|1+𝐵𝐾|]为闭环系统的拓扑李雅普诺夫指数。该问题存在称为尖点障碍的结构性难题:最优增益𝐾*始终使噪声奇点𝑏ₛᵢₙg(𝐾)=−1/𝐾位于噪声支撑集内部。在这一奇异最优解处,策略梯度仅以柯西主值存在,非勒贝格可积,单样本梯度估计器方差无穷。标准一阶随机优化分析在此失效,单纯平滑目标函数亦无法解决此问题。然而,该障碍具有可利用的对称性:柯西核关于极点位移为奇函数,将每条观测与其关于极点的反射配对,即可抵消发散部分。这一单一抵消同时控制了总体曲率、梯度估计方差及噪声密度估计带来的偏差。结合闭式单步梯度预言机,我们证明:投影小批量策略梯度算法,从稳定区域任意紧集初始化,当噪声密度已知时,总样本复杂度为𝑂̃(1/η);当需估计噪声密度且其为𝐶^𝑠光滑(𝑠≥2)时,复杂度为𝑂̃(η^{-(2𝑠+1)/(2𝑠)})。
原文摘要 · Abstract (English)
We study the sample complexity of policy gradient for log-growth control -- the problem of learning, from observed state transitions, a feedback gain that optimally stabilizes a scalar linear system driven through a multiplicative-noise actuation channel. The objective $J(K) = \mathbb{E}[\log|1+BK|]$ is the top Lyapunov exponent of the closed loop. This problem carries a structural difficulty we call the cusp obstruction: the optimal gain $K^*$ always places the noise singularity $b_{\rm sing}(K) = -1/K$ in the interior of the support. At this singular optimum the policy gradient exists only as a Cauchy principal value, not as a Lebesgue integral, and the natural single-sample gradient estimator has infinite variance. Standard first-order stochastic-optimization analysis is thus inapplicable at the optimum, and merely smoothing the objective does not resolve the difficulty. The obstruction, however, has an exploitable symmetry: the Cauchy kernel is an odd function of the displacement from the moving pole, so pairing each observation with its reflection through the pole cancels the divergent part. This one cancellation simultaneously controls the population curvature, the gradient-estimator variance, and the bias incurred when the noise density is estimated. Combining these bounds with a closed-form single-transition gradient oracle, we prove that projected mini-batch policy gradient, initialized in any compact subset of the stabilizing region, attains total sample complexity $\tilde{O}(1/η)$ when the noise density is known and $\tilde{O}(η^{-(2s+1)/(2s)})$ when it must be estimated, for $C^s$ noise densities with $s \geq 2$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。