arXiv:2604.03540cs.RO2026-04中稿 · the 34th ACM Inter…被引 5

让机器人控制模型一次推理完成,速度提升百倍。

Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control

  • 用固定点漂移机制把多步优化融入训练,推理只需一次网络计算
  • 在12个任务上成功率从79%提至83%,推理次数从100次减到1次
  • 适合高频率实时控制,尤其物理机器人部署场景

扩散策略能有效建模机器人操作的多模态动作分布,但其迭代去噪过程需数十至数百次网络函数评估(NFE)才能生成一次控制预测,限制了其在高频闭环控制和在线强化学习中的应用。本文提出两阶段框架,实现原生单步生成策略:首先,基于漂移的策略(DBP)通过固定点漂移目标将修正动态内化至模型参数,设计上仅需一次网络评估即可生成多模态动作块;其次,基于漂移的策略优化(DBPO)在预训练主干上添加随机接口,提供精确的条件轨迹似然,支持类似PPO的在线更新,同时保持1-NFE部署。在12任务扩散策略套件上,DBP将平均成功率从0.79提升至0.83,推理次数从100次降至1次。在37个点云操作任务中,DBP平均成功率达88.4%,超过领先的一次评估基线OMP的82.3%。DBPO进一步在RoboMimic和D4RL上实现了稳定在线微调。在物理双臂UR5平台上,DBP实现123/150成功(82%),端到端延迟均值为9.5毫秒,优于MP1在相同设置下的89/150成功(59%)。代码已开源。

原文摘要 · Abstract (English)

Diffusion policies effectively model multimodal action distributions for robotic manipulation, but their iterative denoising requires tens to hundreds of network function evaluations (NFEs) for each control prediction, limiting their applicability to high-frequency closed-loop control and online reinforcement learning (RL). We present a two-stage framework for native one-step generative policies that transfers iterative refinement from inference to training. First, Drift-Based Policy (DBP) uses a fixed-point drifting objective to internalize corrective dynamics into the model parameters, producing multimodal action chunks with a single network evaluation by design. Second, Drift-Based Policy Optimization (DBPO) augments the pretrained backbone with a stochastic interface that provides exact conditional rollout likelihoods for PPO-style on-policy updates while preserving 1-NFE deployment. On the 12-task Diffusion Policy suite, DBP improves the average success rate from 0.79 to 0.83 while reducing inference from 100 NFEs to 1. Across 37 point-cloud manipulation tasks, DBP achieves an average success rate of 88.4%, surpassing the leading 1-NFE baseline OMP at 82.3%. DBPO further improves pretrained one-step policies through stable online fine-tuning on RoboMimic and D4RL. On a physical dual-arm UR5 platform, DBP achieves 123/150 successes (82%) with an average end-to-end latency of 9.5 ms, compared with MP1's 89/150 successes (59%) under the same setup. Code is available at https://github.com/YuxuanGao0822/DBPO.

机器人控制扩散模型实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。