用分层视觉模型前几阶段实现低成本高精度人体建模
On the Use of Hierarchical Vision Foundation Models for Low-Cost Human Mesh Recovery and Pose Estimation
- 截取分层视觉模型前2-3阶段作为编码器，降低计算开销
- 仅用部分层级即达全模型性能，参数量减少40%以上
- 适合移动端或实时应用的人体姿态与网格重建任务
本文旨在构建简单高效的真人网格恢复（HMR）与姿态估计（HPE）模型。现有先进HMR方法如HMR2.0依赖大型非分层视觉变压器编码器，继承自ViTPose等HPE模型。为建立不同计算预算下的基线，我们先构建了三个轻量级的HMR2.0变体，适配对应的ViTPose模型。此外，我们提出利用分层视觉基础模型（如Swin Transformer、GroupMixFormer、VMamba）的早期阶段作为编码器。该设计基于观察：分层模型中间阶段的特征图分辨率与非分层模型相当或更高。我们对27个基于分层VFMs的HMR和HPE模型进行了全面评估，发现仅使用前两个或三个阶段即可达到与完整模型相当的性能。更重要的是，这些截断模型在精度与效率之间展现出更优的权衡，优于现有轻量级方案。代码已开源：https://github.com/nttcom/TruncHierVFM。
原文摘要 · Abstract (English)
In this work, we aim to develop simple and efficient models for human mesh recovery (HMR) and its predecessor task, human pose estimation (HPE). State-of-the-art HMR methods, such as HMR2.0 and its successors, rely on large, non-hierarchical vision transformers as encoders, which are inherited from the corresponding HPE models like ViTPose. To establish baselines across varying computational budgets, we first construct three lightweight HMR2.0 variants by adapting the corresponding ViTPose models. In addition, we propose leveraging the early stages of hierarchical vision foundation models (VFMs), including Swin Transformer, GroupMixFormer, and VMamba, as encoders. This design is motivated by the observation that intermediate stages of hierarchical VFMs produce feature maps with resolutions comparable to or higher than those of non-hierarchical counterparts. We conduct a comprehensive evaluation of 27 hierarchical-VFM-based HMR and HPE models, demonstrating that using only the first two or three stages achieves performance on par with full-stage models. Moreover, we show that the resulting truncated models exhibit better trade-offs between accuracy and computational efficiency compared to existing lightweight alternatives. The source code is available at https://github.com/nttcom/TruncHierVFM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。