精确的评价网络可显著降低强化学习方差，提升收敛速度。
Refined Analysis of Entropy-Regularized Actor-Critic
- 用精确的批评者作基线，实现强方差缩减
- 达到与确定性策略梯度相当的采样效率，仅需$ ilde{O}("log(1/ε)$)样本
- 建议先训练再更新批评者，适合追求高效收敛的研究者
本文研究熵正则化、有限、折扣环境下的演员-评论家方法中评论家的作用。当评论家精确时，将其作为基线是一种强方差减少方法。此时，使用随机梯度的演员-评论家方法可达到与确定性策略梯度相同的采样复杂度，以$ ilde{O}("log(1/ε)$)次采样获得$ε$-最优正则化价值。实践中评论家与演员同步学习，此时演员更新的方差受评论家的方差和偏差影响。当评论家误差足够小时，方差缩减和快速收敛特性仍能保持。这表明应先独立训练评论家，并在每次演员更新后保持其最新，强调了准确评论家估计在演员-评论家方法中的关键作用。
原文摘要 · Abstract (English)
In this paper, we study the role of the critic in actor--critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the latter as a baseline is a variance-reduction method in a strong sense. In this case, actor--critic with stochastic gradients matches the sample complexity of deterministic policy gradient, reaching an $ε$-optimal regularized value with $\tilde{O}(\log(1/ε))$ samples. In practice, the critic is learned alongside the actor: the variance of the actor update is then influenced by the critic's variance and bias. Specifically, when the critic has a sufficiently small error, the variance reduction and rapid convergence are preserved. This suggests to learn the critic first, keeping it up to date after each actor update, underscoring the crucial role of accurate critic estimation in actor--critic methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。