arXiv:2608.27507cs.LGcs.AI2026-08

通过边际覆盖奖励减少并行策略的冗余探索,提升状态空间覆盖效率。

Marginal Coverage Credit Reduces Redundant Exploration in Parallel State-Entropy Optimization

论文配图:Marginal Coverage Credit Reduces Redundant Exploration in Parallel State-Entropy Optimization
图 1 · 摘自论文原文
  • 基于留一法覆盖度计算每条策略的贡献值,实现精准奖励分配。
  • 在7个公开基准和原论文环境上均显著提升团队状态熵与覆盖度。
  • 适合需要高效探索离散状态空间的强化学习任务,尤其关注互补性覆盖。

并行状态熵最大化策略梯度(PGPSE)通过在相同环境的多个副本中训练独立参数化的策略来扩展状态空间覆盖。然而,其团队熵得分仅衡量整体探索程度,无法识别哪些策略提供了非冗余覆盖。本文提出边际覆盖信用机制(MCC-PGPSE),结合留一法覆盖评估与状态归属专业化,估算各策略的专属贡献。MCC-PGPSE保持原有联合目标,将非负辅助奖励按贡献重新分配,不改变总质量,旨在抑制重复访问、促进互补覆盖。在受控环境、7个公开离散状态基准以及原始PGPSE中的房间与迷宫设置中测试,所有场景下相比熵基线,MCC-PGPSE均实现正向的归一化团队状态熵与状态支持提升。受控任务对比和固定套件聚合结果具有统计显著性,五种子实验的原协议比较方向一致。消融实验与信用对齐控制表明,主要增益来自留一法覆盖度而非非均匀加权、信用错配或神经新颖性。结果支持以贡献条件化的方式分配辅助奖励,是提升离散状态空间中并行策略互补覆盖的有效可解释方法。

原文摘要 · Abstract (English)

Policy Gradient for Parallel State Entropy maximization (PGPSE) expands state-space coverage by training independently parameterized policies in replicated copies of the same environment. However, its pooled team-entropy score measures only collective exploration and cannot identify policies that contribute non-redundant coverage. We introduce Marginal Coverage Credit for PGPSE (MCC-PGPSE), which combines leave-one-policy-out coverage with state-owner specialization to estimate policy-specific credit. MCC-PGPSE preserves PGPSE's pooled objective and redistributes non-negative auxiliary intrinsic rewards according to these credits without changing their total mass. This redistribution is designed to discourage redundant visitation and promote complementary coverage. We evaluated MCC-PGPSE in controlled environments, seven public discrete-state benchmarks, and representative Room and Maze settings from the original PGPSE protocol. Across all tested settings, MCC-PGPSE produced positive final window gains in normalized team state entropy and state support over the Entropy baseline. Controlled-task comparisons and the fixed-suite public aggregate were significant, whereas five-seed original-protocol comparisons were directionally consistent. Ablations and credit alignment controls indicate that most gains arise from leave-one-policy-out coverage rather than non-uniform weighting, mismatched credit, or neural novelty alone. These results support contribution-conditioned auxiliary reward allocation as an interpretable approach to improving complementary coverage among parallel policies in discrete state spaces.

强化学习状态覆盖并行策略奖励设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。