将视觉理解与轨迹生成分离,提升多臂协作的精度与泛化能力。
Latent Diffusion Policy: Shaping Latent Spaces for Diffusion-Based Robotic Manipulation

- 先用条件变分自编码器提取场景特征,再在压缩的隐空间生成动作
- 在RoboTwin 2.0上比DP3成功率高,且可直接迁移到真实双臂机器人
- 引入轻量级重建FID评估隐空间质量,无需执行任务即可预测成功
基于扩散模型的视觉运动策略直接在原始动作空间中操作,将场景理解与轨迹生成耦合在一个去噪过程中。由此产生的速度场需同时编码场景信息并生成精确轨迹,增加了学习复杂度,限制了对多臂精密时序协调任务的表现。为简化这一联合学习问题,我们提出隐空间扩散策略(LDP),采用两阶段框架,在精心设计的隐空间中进行流匹配。通过将场景理解融入观察条件化的CVAE编码器,LDP集中每个观测的条件分布。因此,流模型无需隐式解析依赖场景的结构;相反,它在预集中的分布内生成,具有更平滑的速度场,从而从有限示范中更易学习。此外,为捕捉隐令牌间的时序依赖,LDP采用逐令牌扩散强制训练,并使用阶梯式推理采样以解决分布不匹配问题。我们还提出重建FID(rFID)作为轻量级代理指标,仅基于隐空间统计量即可预测下游任务成功率。在RoboTwin 2.0的协调密集型任务中,LDP显著优于DP3,并有效迁移至真实世界双臂部署。
原文摘要 · Abstract (English)
Diffusion-based visuomotor policies operating directly in raw action spaces conflate scene comprehension with trajectory generation within a single denoising process. The resulting velocity field must simultaneously encode scene information and generate precise trajectories, increasing learning complexity and limiting performance on tasks demanding precise temporal coordination across multiple arms. To simplify this joint learning problem, we introduce Latent Diffusion Policy (LDP), a two-stage framework performing flow matching in a deliberately shaped latent space. By absorbing scene understanding into an observation-conditioned CVAE encoder, LDP concentrates the conditional distribution of each observation. Consequently, the flow model avoids implicitly resolving scene-dependent structures; instead, it generates within a pre-concentrated distribution featuring a smoother velocity field, simplifying learning from limited demonstrations. Furthermore, to capture temporal dependencies among latent tokens, LDP trains with per-token diffusion forcing and employs staircase inference sampling to resolve the resulting distributional mismatch. We also propose reconstruction FID (rFID) as a lightweight proxy predicting downstream task success solely from latent space statistics. On coordination-intensive tasks from RoboTwin 2.0, LDP outperforms DP3 by a substantial margin and transfers effectively to real-world bimanual deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。