无需训练即可实现视频中人体网格的时序一致恢复
SAM-Body4D: Training-Free 4D Human Body Mesh Recovery from Videos
- 利用视频中人体连续性，通过提示分割生成身份一致的掩码块
- 引入遮挡感知模块修复缺失区域，提升遮挡下的恢复鲁棒性
- 适合作为无需微调的视频人体重建工具，支持多人体并行
人体网格恢复（HMR）旨在从2D观测中重建3D人体姿态与形状，是真实场景下以人为中心理解的基础。尽管近期基于图像的HMR方法如SAM 3D Body在野外图像上表现稳健，但在视频应用中依赖逐帧推理，导致时间不一致且在遮挡下性能下降。本文提出SAM-Body4D，一种无需训练的视频级4D人体网格恢复框架，通过利用视频中的人体连续性解决上述问题。首先使用可提示视频分割模型生成身份一致的掩码块，再通过遮挡感知模块修复缺失区域。优化后的掩码块引导SAM 3D Body生成一致的全身网格轨迹，同时采用基于填充的并行策略实现高效多人体推理。实验表明，SAM-Body4D在挑战性野外视频中显著提升了时间稳定性与遮挡鲁棒性，且无需任何重训练。代码与演示已开源。
原文摘要 · Abstract (English)
Human Mesh Recovery (HMR) aims to reconstruct 3D human pose and shape from 2D observations and is fundamental to human-centric understanding in real-world scenarios. While recent image-based HMR methods such as SAM 3D Body achieve strong robustness on in-the-wild images, they rely on per-frame inference when applied to videos, leading to temporal inconsistency and degraded performance under occlusions. We address these issues without extra training by leveraging the inherent human continuity in videos. We propose SAM-Body4D, a training-free framework for temporally consistent and occlusion-robust HMR from videos. We first generate identity-consistent masklets using a promptable video segmentation model, then refine them with an Occlusion-Aware module to recover missing regions. The refined masklets guide SAM 3D Body to produce consistent full-body mesh trajectories, while a padding-based parallel strategy enables efficient multi-human inference. Experimental results demonstrate that SAM-Body4D achieves improved temporal stability and robustness in challenging in-the-wild videos, without any retraining. Our code and demo are available at: https://github.com/gaomingqi/sam-body4d.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。