用视觉掩码修复复杂动作，让视频生成的动作更符合物理规律。
A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions
- 通过视频掩码和运动上下文定位并修正错误动作片段。
- 在真实复杂场景下提升动作物理合理性，优于现有方法。
- 可直接插入现有流程，适合处理高难度野外动作数据。
从视频中提取符合物理规律的三维人体动作为关键任务。尽管已有基于仿真的动作模仿方法能提升单目视频估计的日常动作物理质量，但将其拓展至高难度动作仍具挑战，主要源于视频捕捉结果中的错误动作片段以及高难度动作建模的固有复杂性。为此，我们利用分割在局部定位人体的优势，提出基于掩码的动作修正模块（MCM），通过运动上下文与视频掩码修复缺陷动作，生成利于模仿的高质量动作；同时设计基于物理的动作迁移模块（PTM），采用预训练+自适应策略实现动作模仿，在保持真实感的同时有效处理真实场景下的高难度动作。该方法为即插即用模块，可对视频动作捕捉结果进行物理级精炼，包括高难度野外动作。为验证效果，我们构建了具有挑战性的野外测试集，建立新基准。实验表明，该方法在新基准及现有公开数据集上均表现优异。
原文摘要 · Abstract (English)
Extracting physically plausible 3D human motion from videos is a critical task. Although existing simulation-based motion imitation methods can enhance the physical quality of daily motions estimated from monocular video capture, extending this capability to high-difficulty motions remains an open challenge. This can be attributed to some flawed motion clips in video-based motion capture results and the inherent complexity in modeling high-difficulty motions. Therefore, sensing the advantage of segmentation in localizing human body, we introduce a mask-based motion correction module (MCM) that leverages motion context and video mask to repair flawed motions, producing imitation-friendly motions; and propose a physics-based motion transfer module (PTM), which employs a pretrain and adapt approach for motion imitation, improving physical plausibility with the ability to handle in-the-wild and challenging motions. Our approach is designed as a plug-and-play module to physically refine the video motion capture results, including high-difficulty in-the-wild motions. Finally, to validate our approach, we collected a challenging in-the-wild test set to establish a benchmark, and our method has demonstrated effectiveness on both the new benchmark and existing public datasets.https://physicalmotionrestoration.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。