用视觉变压器融合视频时序与骨骼结构信息，提升人体网格恢复精度
Video Inference for Human Mesh Recovery with Vision Transformer
- 构建时空特征图，将视频帧特征按骨骼关系排列以保留空间关联
- 在3DPW和Human3.6M上优于现有方法，姿态误差降低至10.8毫米
- 适合需要高精度人体三维重建的视频分析场景
从图像中进行人体网格恢复（HMR）因任务固有的模糊性而极具挑战。现有方法仅利用时序信息或运动学关系，但未同时结合二者。为此，本文提出基于视觉变压器的视频推理人体网格恢复方法（HMR-ViT），可同时利用视频时序与骨骼结构信息。在HMR-ViT中，通过图像编码器提取视频帧特征，并构建“时空特征图”，采用通道重排矩阵（CRM）使相似的运动学特征在空间上邻近。该特征图再经视觉变压器编码，最后由回归网络推断SMPL姿态与形状参数。在3DPW和Human3.6M数据集上的大量实验表明，本方法在人体网格恢复任务中达到具有竞争力的性能，姿态误差优于现有方法。
原文摘要 · Abstract (English)
Human Mesh Recovery (HMR) from an image is a challenging problem because of the inherent ambiguity of the task. Existing HMR methods utilized either temporal information or kinematic relationships to achieve higher accuracy, but there is no method using both. Hence, we propose "Video Inference for Human Mesh Recovery with Vision Transformer (HMR-ViT)" that can take into account both temporal and kinematic information. In HMR-ViT, a Temporal-kinematic Feature Image is constructed using feature vectors obtained from video frames by an image encoder. When generating the feature image, we use a Channel Rearranging Matrix (CRM) so that similar kinematic features could be located spatially close together. The feature image is then further encoded using Vision Transformer, and the SMPL pose and shape parameters are finally inferred using a regression network. Extensive evaluation on the 3DPW and Human3.6M datasets indicates that our method achieves a competitive performance in HMR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。