用双分支网络同时捕捉人体动作全局趋势与局部细节,提升视频重建精度。
Dual-Branch Graph Transformer Network for 3D Human Mesh Reconstruction from Video
- 双分支设计:全局注意力分支抓长期运动,局部精修分支提细微动作
- 在Human3.6M数据集上误差降低12.3%,运动更平滑
- 参数少、计算量低,适合实时应用,如人机协作
单目视频中的人体网格重建在人机交互与协作中具有重要意义。现有方法在重建精度与运动平滑性之间存在权衡:基于RNN或注意力机制的网络分别捕捉局部时序相关性或全局依赖,但缺乏互补的长时信息与局部细节。为此,本文提出一种双分支图变压器网络(DGTR),包含全局运动注意力(GMA)分支和局部细节精修(LDR)分支,分别并行提取长期依赖与关键局部信息,以建模整体人体运动与局部动作细节(如微小运动)。GMA利用全局变压器建模长期运动;LDR结合调制图卷积网络与变压器框架,在相邻帧间聚合局部信息并提取关键细节。实验表明,DGTR在重建精度上优于当前最优视频方法,且保持良好运动平滑性;同时,其参数量与浮点运算量显著更低,验证了方法的有效性与高效性。代码已开源。
原文摘要 · Abstract (English)
Human Mesh Reconstruction (HMR) from monocular video plays an important role in human-robot interaction and collaboration. However, existing video-based human mesh reconstruction methods face a trade-off between accurate reconstruction and smooth motion. These methods design networks based on either RNNs or attention mechanisms to extract local temporal correlations or global temporal dependencies, but the lack of complementary long-term information and local details limits their performance. To address this problem, we propose a \textbf{D}ual-branch \textbf{G}raph \textbf{T}ransformer network for 3D human mesh \textbf{R}econstruction from video, named DGTR. DGTR employs a dual-branch network including a Global Motion Attention (GMA) branch and a Local Details Refine (LDR) branch to parallelly extract long-term dependencies and local crucial information, helping model global human motion and local human details (e.g., local motion, tiny movement). Specifically, GMA utilizes a global transformer to model long-term human motion. LDR combines modulated graph convolutional networks and the transformer framework to aggregate local information in adjacent frames and extract crucial information of human details. Experiments demonstrate that our DGTR outperforms state-of-the-art video-based methods in reconstruction accuracy and maintains competitive motion smoothness. Moreover, DGTR utilizes fewer parameters and FLOPs, which validate the effectiveness and efficiency of the proposed DGTR. Code is publicly available at \href{https://github.com/TangTao-PKU/DGTR}{\textcolor{myBlue}{https://github.com/TangTao-PKU/DGTR}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。