实时操控中高效融合3D视觉先验,提升成功率并降低延迟。
R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation
- 异步快慢协同模块:仅在关键帧调用大模型,中间帧由轻量网络预测
- 多视图融合显式引入相机内外参,提升特征一致性,成功率提升51.4%
- 可插拔设计,适合需实时性的机器人操控任务
具身操作需要精确的物体3D理解及其空间关系以规划和执行接触丰富的动作。尽管大规模3D视觉模型提供了强大的先验知识,但其计算开销导致实时控制延迟过高。本文提出实时3D感知策略(R3DP),在不牺牲实时性能的前提下整合强大3D先验。核心创新是异步快慢协同模块,通过仅在稀疏关键帧查询预训练慢速系统(VGGT),同时使用轻量级时序特征预测网络(TFPNet)预测所有中间帧特征,利用历史数据捕捉时间相关性,从而提升任务成功率。此外,为实现更有效的多视图融合,引入多视图特征融合器(MVFF),显式结合相机内参与外参进行特征聚合。R3DP为将大型模型集成至实时推理系统提供即插即用方案。在多种视觉配置下评估,相比单视图和多视图基线,平均成功率分别提升32.9%和51.4%;通过解耦重3D推理与策略执行,相比直接集成的DP+VGGT,推理时间减少44.8%。
原文摘要 · Abstract (English)
Embodied manipulation requires accurate 3D understanding of objects and their spatial relations to plan and execute contact-rich actions. While large-scale 3D vision models provide strong priors, their computational cost incurs prohibitive latency for real-time control. We propose Real-time 3D-aware Policy (R3DP), which integrates powerful 3D priors into manipulation policies without sacrificing real-time performance. A core innovation of R3DP is the asynchronous fast-slow collaboration module, which seamlessly integrates large-scale 3D priors into the policy without compromising real-time performance. The system maintains real-time efficiency by querying the pre-trained slow system (VGGT) only on sparse key frames, while simultaneously employing a lightweight Temporal Feature Prediction Network (TFPNet) to predict features for all intermediate frames. By leveraging historical data to exploit temporal correlations, TFPNet explicitly improves task success rates through consistent feature estimation. Additionally, to enable more effective multi-view fusion, we introduce a Multi-View Feature Fuser (MVFF) that aggregates features across views by explicitly incorporating camera intrinsics and extrinsics. R3DP offers a plug-and-play solution for integrating large models into real-time inference systems. We evaluate R3DP against multiple baselines across different visual configurations. R3DP effectively harnesses large-scale 3D priors to achieve superior results, outperforming single-view and multi-view DP by 32.9% and 51.4% in average success rate, respectively. Furthermore, by decoupling heavy 3D reasoning from policy execution, R3DP achieves a 44.8% reduction in inference time compared to a naive DP+VGGT integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。