熵的测量位置影响策略几何，决定智能体是否倾向边界动作。
Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO
- 在策略优化中，熵的测量位置（隐变量或执行动作）改变策略的均值分布
- 使用执行动作熵时，71.4%的动作接近边界；而用隐变量熵时，该比例高达89.1%
- 即使移除方差影响，仍有多达83.8%的动作靠近边界，说明问题出在均值设计
许多连续控制策略被优化为无界高斯分布，再映射到有界动作空间。我们发现，熵的测量位置会影响近端策略优化（PPO）所学得的策略几何结构。在80肌肉的MyoLeg任务中，采用截断高斯分布时，89.07%的动作位于边界5%范围内。相同状态下分析表明，这并非仅由方差导致：即使方差设为零，仍有83.83%的动作靠近边界，而82.12%的状态条件均值位于可执行区间外。用tanh映射替代截断无法消除高方差区域。对隐变量熵H(u)，熵损失对均值梯度为零，对方差有恒定上升梯度；对执行动作熵H(a)，变换雅可比矩阵在均值上施加向内梯度。在三个匹配的MyoLeg种子实验中，近边界占据率分别为71.42%、29.76%和18.83%，分别对应隐变量熵、无熵、执行动作熵。38维狗站立任务复现了均值几何的顺序，并在共享状态评估和1%至10%边界裕度下保持稳定。直接施加均值惩罚可达到甚至超过执行动作熵带来的中心化效果，表明内部均值并非执行熵独有。但匹配的均值几何可与显著不同的方差和回报共存。因此，熵的测量空间是耦合的均值-方差设计选择，任务回报本身不足以刻画有界策略的几何结构。
原文摘要 · Abstract (English)
Many continuous-control policies are optimized as unbounded Gaussians and then mapped into bounded actions. We show that where entropy is measured changes the policy geometry learned by proximal policy optimization (PPO). In an 80-muscle MyoLeg task, a clipped Gaussian executes 89.07% of actions within 5% of a bound. A same-state decomposition shows that this is not due to variance alone: setting variance to zero still leaves 83.83% of actions near a bound, while 82.12% of state-conditioned means lie outside the executable interval. Replacing clipping with a tanh map does not remove the high-variance regime. For latent Gaussian entropy H(u), the entropy loss has zero gradient with respect to the mean and a constant variance-increasing gradient. For executed-action entropy H(a), the transform Jacobian adds an inward gradient on the mean. Across three matched MyoLeg seeds, near-boundary occupancy is 71.42%, 29.76%, and 18.83% under latent entropy, no entropy, and executed-action entropy. A 38-dimensional Dog-Stand replication with an independent CleanRL-based PPO implementation reproduces the ordering in mean geometry, which also survives shared-state evaluation and boundary margins from 1% to 10%. Direct mean penalties can match or exceed the centering produced by H(a), showing that interior means are not unique to executed entropy. However, matched mean geometry can coexist with substantially different variance and return. Entropy measurement space is therefore a coupled mean-variance design choice, and task return alone does not characterize bounded-policy geometry.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。