通过一步流策略实现高保真隐空间策略优化,无需敏感调参。
Latent Policy Steering through One-Step Flow Policies
- 用可微一步均值流策略传递原始动作空间的Q梯度,直接优化隐空间策略。
- 在OGBench和真实机器人任务上超越行为克隆与主流隐空间引导方法。
- 无需代理隐空间评判器,开箱即用,对超参数不敏感。
离线强化学习允许机器人在不进行危险探索的情况下从离线数据集中学习。然而,离线RL的性能通常受限于两个方面的权衡:(1) 回报最大化,可能导致策略超出数据集支持范围;(2) 行为约束,通常需要敏感的超参数调节。隐空间引导提供了一种结构化方式以保持策略在数据集支持范围内,但现有离线方法普遍采用间接蒸馏学习的隐空间评判器来近似动作价值,易丢失信息并阻碍收敛。本文提出隐空间策略引导(LPS),通过可微的一步均值流策略将原始动作空间的Q梯度反向传播,更新隐动作空间的策略网络。该方法摒弃了代理隐空间评判器,使原始动作空间的评判器能够端到端指导隐空间优化,同时一步均值流策略作为受约束的生成先验。这种解耦设计带来稳健的方法,几乎无需调参即可直接使用。在OGBench和真实机器人任务中,LPS实现了当前最优性能,并持续优于行为克隆和强基准的隐空间引导方法。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration. Yet, offline RL's performance often hinges on a brittle trade-off between (1) return maximization, which can push policies outside the dataset support, and (2) behavioral constraints, which typically require sensitive hyperparameter tuning. Latent steering offers a structural way to stay within the dataset support during RL, but existing offline adaptations commonly approximate action values using latent-space critics learned via indirect distillation, which can lose information and hinder convergence. We propose Latent Policy Steering (LPS), which enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. By eliminating proxy latent critics, LPS allows an original-action-space critic to guide end-to-end latent-space optimization, while the one-step MeanFlow policy serves as a behavior-constrained generative prior. This decoupling yields a robust method that works out-of-the-box with minimal tuning. Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。