用统一扩散模型实现实时灵巧操作仿真，支持手部动作迁移与高保真交互。
DexSIM: Real-time Dexterous Simulation with Unified Causal Video Diffusion
- 双向扩散模型联合编码手部轨迹与视频，用高斯热图提升手部表征精度
- 基于滚动预测的自回归训练结合空间缓存，实现长期空间一致性与3D感知
- 支持实时交互（15.24 FPS），在像素、语义和手部投影上优于基线
视频扩散模型的进展推动了物理世界模拟的广泛应用，但手物交互模拟仍较少被探索。我们提出DexSIM，一个面向实时灵巧操作的仿真框架。现有方法在导航任务中表现良好，但在灵巧操作方面受限于缺乏实时交互性、长期空间一致性和记忆能力。DexSIM采用两阶段训练：首先通过联合嵌入手部动作轨迹与视频到统一特征空间，训练双向视频扩散模型，并使用高斯热图编码提升手部表示精度；其次通过基于滚动预测的自回归训练，引入更新的空间缓存作为注意力汇点，实现空间记忆与长期一致性。DexSIM在像素相似性、语义相似性、运动保真度和手部投影准确性上均优于基线，支持手部动作迁移新应用，运行速度达15.24 FPS，实现真正实时交互。
原文摘要 · Abstract (English)
Recent progress of video diffusion models have enabled extensive simulation of the physical world. While simulation with hand object interaction has been less explored. We propose DexSIM, a dexterous simulation framework for simulating dexterous manipulation in real-time. While previous works utilizing video diffusion and 3D reconstruction focus on navigation, dexterous manipulation has been limited while it has extensive applications for creating interactive experiences with the simulated world and for generating synthetic data for robotics. Existing methods lack real-time interactivity and long-term spatial consistency and memory. We propose a 2-stage training framework for DexSIM. First we train a bi-directional video diffusion model by jointly embedding the hand action trajectory and video in a unified feature space. We utilize gaussian heatmap hand encoding for more accurate hand representation. Then we conduct a roll-out based autoregressive training with updated spatial cache as attention sink for spatial memory, which improves long-term consistency and 3D aware dexterous manipulation simulation. DexSIM outperforms the baseline on pixel and semantic similarity, motion fidelity, and hand projection accuracy. It also allows new applications such as hand motion transfer and runs at 15.24 FPS real-time interactivity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。