用裁剪目标优化提升生成策略的稳定性和效率
Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning
- 将策略优化转化为后验推断问题，避免显式似然估计
- 在7个仿真和4个真实任务中实现96.7%成功率，防止策略崩溃
- 适用于大模型微调，无需修改架构，适合机器人控制场景
表达性强的生成模型通过捕捉长时间轨迹上的复杂多模态动作分布，推动了机器人操作的发展。然而，基于强化学习（RL）进行微调仍面临不稳定与样本效率低的问题。本文提出后验优化裁剪目标（POCO），一种面向时间动作片段的原理性强化学习框架。通过期望最大化过程，POCO将奖励加权的隐式后验提炼到策略中，无需进行似然估计。此外，POCO采用离线到在线范式，将在线探索锚定于预训练先验，并具备模型无关设计，可直接微调大型视觉-语言-动作（VLA）模型而无需结构修改。在7个仿真基准和4个接触丰富的现实任务上评估显示，POCO有效防止灾难性策略崩溃，性能优于现有最先进方法，在真实任务中达到96.7%的成功率。视频展示见项目主页 https://cccedric.github.io/poco/
原文摘要 · Abstract (English)
Expressive generative models have advanced robotic manipulation by capturing complex, multi-modal action distributions over temporally extended trajectories. However, fine-tuning these policies via RL remains challenging due to instability and sample inefficiency. We introduce Posterior Optimization with Clipped Objective (POCO), a principled RL framework that formulates policy improvement as a posterior inference problem tailored for temporal action chunks. Through an Expectation-Maximization procedure, POCO distills a reward-weighted implicit posterior into the policy without likelihood estimation. Furthermore, POCO adopts an offline-to-online paradigm that anchors online exploration to pre-trained priors, and its model-agnostic design scales to fine-tune large VLA models without architectural modifications. Evaluations across 7 simulation benchmarks and 4 contact-rich real-world tasks demonstrate that POCO prevents catastrophic policy collapse, outperforms SOTA baselines, and achieves a 96.7% success rate on real-world tasks. Videos are available at our project website https://cccedric.github.io/poco/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。