arXiv:2501.09782cs.CVcs.GR2025-01TPAMI被引 45

通过1000万数据和大模型,打造通用人体姿态与形状估计基础模型。

SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimation

  • 用40个数据集系统筛选,整合1000万训练样本提升泛化能力。
  • 采用视觉变换器架构,在7个基准上达到领先性能。
  • 模型可微调为专业领域模型,适合多场景动作捕捉应用。

表情化人体姿态与形状估计(EHPS)融合身体、手部和面部动作捕捉,应用广泛。尽管进展显著,现有方法多聚焦于特定数据集上的创新架构设计。本文探索将EHPS向通用基础模型方向规模化。在数据层面,系统分析40个EHPS数据集,涵盖多样场景,最终整合超过1000万来自异构数据源的训练实例,实现性能跃升并趋于饱和。在模型层面,采用视觉变换器(最大至ViT-Huge),基于两种极简架构:SMPLer-X(含手/面定位中间步骤)与更简化的SMPLest-X,以排除算法设计干扰。结合海量数据与大模型,基础模型在多个测试基准上表现优异,且具备良好跨场景迁移能力。微调策略可将其转化为专业模型,进一步提升性能。在AGORA、UBody、EgoBody及自建SynHand数据集等7个基准上持续取得最优结果。(代码见:https://github.com/wqyin/SMPLest-X)

原文摘要 · Abstract (English)

Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications. Despite encouraging progress, current state-of-the-art methods focus on training innovative architectural designs on confined datasets. In this work, we investigate the impact of scaling up EHPS towards a family of generalist foundation models. 1) For data scaling, we perform a systematic investigation on 40 EHPS datasets, encompassing a wide range of scenarios that a model trained on any single dataset cannot handle. More importantly, capitalizing on insights obtained from the extensive benchmarking process, we optimize our training scheme and select datasets that lead to a significant leap in EHPS capabilities. Ultimately, we achieve diminishing returns at 10M training instances from diverse data sources. 2) For model scaling, we take advantage of vision transformers (up to ViT-Huge as the backbone) to study the scaling law of model sizes in EHPS. To exclude the influence of algorithmic design, we base our experiments on two minimalist architectures: SMPLer-X, which consists of an intermediate step for hand and face localization, and SMPLest-X, an even simpler version that reduces the network to its bare essentials and highlights significant advances in the capture of articulated hands. With big data and the large model, the foundation models exhibit strong performance across diverse test benchmarks and excellent transferability to even unseen environments. Moreover, our finetuning strategy turns the generalist into specialist models, allowing them to achieve further performance boosts. Notably, our foundation models consistently deliver state-of-the-art results on seven benchmarks such as AGORA, UBody, EgoBody, and our proposed SynHand dataset for comprehensive hand evaluation. (Code is available at: https://github.com/wqyin/SMPLest-X).

人体估计大模型动作捕捉视觉变换器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。