通过结构化初始化提升离线强化学习在离散动作空间的效率与性能。
Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization
- 先预训练动作结构模型捕捉有效动作分布，再冻结其表示训练轻量策略头。
- 在DM Control基准上平均回报提升39%，收敛速度加快12.8倍。
- 适合需要高效探索大规模离散动作空间的离线强化学习任务。
在离散组合动作空间中进行强化学习需同时选择多个子动作以形成一致组合，现有方法或假设子动作独立（导致无效动作），或联合学习结构与控制（效率低且不稳定）。本文提出结构化策略初始化（SPIN），分两阶段：首先预训练动作结构模型（ASM）以捕捉有效动作流形，随后冻结该表示并训练轻量级策略头进行控制。在具有挑战性的离散DM Control基准上，SPIN相比当前最优方法平均回报提升最高达39%，收敛时间减少最多12.8倍。
原文摘要 · Abstract (English)
Reinforcement learning in discrete combinatorial action spaces requires searching over exponentially many joint actions to simultaneously select multiple sub-actions that form coherent combinations. Existing approaches either simplify policy learning by assuming independence across sub-actions, which often yields incoherent or invalid actions, or attempt to learn action structure and control jointly, which is slow and unstable. We introduce Structured Policy Initialization (SPIN), a two-stage framework that first pre-trains an Action Structure Model (ASM) to capture the manifold of valid actions, then freezes this representation and trains lightweight policy heads for control. On challenging discrete DM Control benchmarks, SPIN improves average return by up to 39% over the state of the art while reducing time to convergence by up to 12.8$\times$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。