通过分阶段探索与优化，让机器人在安全前提下高效学习新技能。
SPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Space
- 先在低维隐空间安全探索，再过渡到原始动作空间避免性能瓶颈
- 在厨房和行走任务上实现超90%的归一化回报，样本效率提升5倍
- 无需轨迹分段，仅需无序状态动作对，适合实际部署
离线到在线强化学习为机器人提供了预训练安全策略并在线微调的前景。然而核心挑战在于：如何在不偏离离线数据行为支持的前提下安全地进行在线探索？现有方法虽利用条件变分自编码器（CVAE）将探索限制在隐空间，但存在固有的利用差距——由解码器重建损失带来的性能天花板。本文提出SPAARS，一种课程学习框架：初期将探索约束在低维隐流形中以实现高效且安全的行为改进，随后无缝转移至原始动作空间，绕过解码器瓶颈。SPAARS有两种实现方式：基于CVAE的版本仅需无序（状态,动作）对，无需轨迹分割；SPAARS-SUPE结合OPAL时序技能预训练，增强探索结构，但需轨迹片段。我们证明了利用差距的上界，确立隐空间策略梯度在方差上优于原始空间探索，并表明在隐空间阶段同时进行行为克隆可直接控制课程过渡稳定性。实验显示，SPAARS-SUPE在kitchen-mixed-v0上达到0.825的归一化回报，优于SUPE的0.75，样本效率提高5倍；独立版SPAARS在hopper-medium-v2和walker2d-medium-v2上分别取得92.7和102.9的归一化回报，显著超过IQL基线（66.3和78.3），验证了无序对CVAE实例化的有效性。
原文摘要 · Abstract (English)
Offline-to-online reinforcement learning (RL) offers a promising paradigm for robotics by pre-training policies on safe, offline demonstrations and fine-tuning them via online interaction. However, a fundamental challenge remains: how to safely explore online without deviating from the behavioral support of the offline data? While recent methods leverage conditional variational autoencoders (CVAEs) to bound exploration within a latent space, they inherently suffer from an exploitation gap -- a performance ceiling imposed by the decoder's reconstruction loss. We introduce SPAARS, a curriculum learning framework that initially constrains exploration to the low-dimensional latent manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck. SPAARS has two instantiations: the CVAE-based variant requires only unordered (s,a) pairs and no trajectory segmentation; SPAARS-SUPE pairs SPAARS with OPAL temporal skill pretraining for stronger exploration structure at the cost of requiring trajectory chunks. We prove an upper bound on the exploitation gap using the Performance Difference Lemma, establish that latent-space policy gradients achieve provable variance reduction over raw-space exploration, and show that concurrent behavioral cloning during the latent phase directly controls curriculum transition stability. Empirically, SPAARS-SUPE achieves 0.825 normalized return on kitchen-mixed-v0 versus 0.75 for SUPE, with 5x better sample efficiency; standalone SPAARS achieves 92.7 and 102.9 normalized return on hopper-medium-v2 and walker2d-medium-v2 respectively, surpassing IQL baselines of 66.3 and 78.3 respectively, confirming the utility of the unordered-pair CVAE instantiation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。