通过状态覆盖匹配,让智能体行为更高效地压缩到低维空间。
Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching
- 用长期状态覆盖替代即时动作匹配,提升行为相似性建模。
- 在多个连续控制任务上,压缩后策略性能损失小于5%。
- 适合研究高效策略表示与无监督行为压缩的学者。
深度强化学习因样本效率低而受限,部分原因在于策略参数空间维度高且存在大量功能冗余。现有基于动作的策略压缩(APC)方法通过生成映射 $g:\mathcal Z \to Θ$ 将高维参数空间 $Θ$ 压缩至低维潜在流形 $\mathcal Z$,但其性能受限于依赖即时动作匹配作为重建损失,该方式对行为相似性建模过于短视,易累积决策误差。为此,本文提出基于状态覆盖的策略压缩(OPC),将行为表征从即时动作匹配转向长期状态空间覆盖率。核心改进包括:(1) 利用信息论唯一性度量构建多样化策略数据集;(2) 设计可全程微分的压缩目标,直接最小化真实与重构混合状态占用分布之间的差异。该机制促使生成模型在潜在空间中围绕真实功能相似性组织,实现广泛行为泛化的同时保持原参数空间的表达能力。我们在多个连续控制基准任务上实证验证了该方法的优势。
原文摘要 · Abstract (English)
Deep Reinforcement Learning (DRL) is widely recognized as sample-inefficient, a limitation attributable in part to the high dimensionality and substantial functional redundancy inherent to the policy parameter space. A recent framework, which we refer to as Action-based Policy Compression (APC), mitigates this issue by compressing the parameter space $Θ$ into a low-dimensional latent manifold $\mathcal Z$ using a learned generative mapping $g:\mathcal Z \to Θ$. However, its performance is severely constrained by relying on immediate action-matching as a reconstruction loss, a myopic proxy for behavioral similarity that suffers from compounding errors across sequential decisions. To overcome this bottleneck, we introduce Occupancy-based Policy Compression (OPC), which enhances APC by shifting behavior representation from immediate action-matching to long-horizon state-space coverage. Specifically, we propose two principal improvements: (1) we curate the dataset generation with an information-theoretic uniqueness metric that delivers a diverse population of policies; and (2) we propose a fully differentiable compression objective that directly minimizes the divergence between the true and reconstructed mixture occupancy distributions. These modifications force the generative model to organize the latent space around true functional similarity, promoting a latent representation that generalizes over a broad spectrum of behaviors while retaining most of the original parameter space's expressivity. Finally, we empirically validate the advantages of our contributions across multiple continuous control benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。