揭示了软策略梯度中逻辑斯蒂更新的自调节机制
Logit Dynamics in Softmax Policy Gradient Methods
- 推导出逻辑斯蒂更新向量的L2范数精确公式
- 更新幅度由动作概率和策略碰撞概率共同决定
- 解释了方法稳定性，适合研究强化学习优化机制者
本文分析了软策略梯度方法中的逻辑斯蒂动态。推导出逻辑斯蒂更新向量的L2范数精确公式：‖Δz‖₂ ∝ √(1−2P_c + C(P))。该公式表明，更新幅度由所选动作的概率P_c和策略的碰撞概率C(P)决定，后者是熵的反相关浓度度量。分析揭示了一种内在的自调节机制，学习强度会随策略置信度自动调节，为这类方法的稳定性和收敛性提供了基础性洞见。
原文摘要 · Abstract (English)
We analyzes the logit dynamics of softmax policy gradient methods. We derive the exact formula for the L2 norm of the logit update vector: $$ \|Δ\mathbf{z}\|_2 \propto \sqrt{1-2P_c + C(P)} $$ This equation demonstrates that update magnitudes are determined by the chosen action's probability ($P_c$) and the policy's collision probability ($C(P)$), a measure of concentration inversely related to entropy. Our analysis reveals an inherent self-regulation mechanism where learning vigor is automatically modulated by policy confidence, providing a foundational insight into the stability and convergence of these methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。