提出两种高效2D人体姿态估计模型,兼顾精度与速度。
Transformers with Joint Tokens and Local-Global Attention for Efficient Human Pose Estimation
- 用可学习关节令牌选择关键身体区域,动态控制计算量
- 在六大数据集上提升精度,计算量降低30%至44%
- 统一多数据集骨骼表示,增强跨场景鲁棒性
卷积神经网络(CNN)和视觉变压器(ViT)在2D人体姿态估计中取得显著进展。然而,在精度、效率和鲁棒性之间保持平衡仍是挑战:CNN计算高效但难以捕捉长距离依赖,而ViT虽擅长建模长程关系,却存在二次计算复杂度问题。本文提出两种基于ViT的模型实现高精度、高效率、强鲁棒性的2D姿态估计。第一种模型EViTPose通过可学习的关节令牌选择并处理最重要的身体块,实现计算效率与精度的灵活权衡,通过调整处理块数即可调节性能。第二种模型UniTransPose结合局部-全局注意力的多尺度变压器编码器与高效的亚像素CNN解码器,在不牺牲速度的前提下提升精度。此外,通过将多个基准数据集的所有关节整合为统一骨架表示,联合训练出在多种场景下表现优异的鲁棒模型,包括姿态变化、光照差异和遮挡情况。在六个基准测试上的实验表明,所提方法显著优于现有最佳模型,且计算效率更高:EViTPose的计算量降低30%至44%(GFLOPs),精度下降仅0%至3.5%;UniTransPose在各数据集上精度提升达0.9%至43.8%。
原文摘要 · Abstract (English)
Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have led to significant progress in 2D body pose estimation. However, achieving a good balance between accuracy, efficiency, and robustness remains a challenge. For instance, CNNs are computationally efficient but struggle with long-range dependencies, while ViTs excel in capturing such dependencies but suffer from quadratic computational complexity. This paper proposes two ViT-based models for accurate, efficient, and robust 2D pose estimation. The first one, EViTPose, operates in a computationally efficient manner without sacrificing accuracy by utilizing learnable joint tokens to select and process a subset of the most important body patches, enabling us to control the trade-off between accuracy and efficiency by changing the number of patches to be processed. The second one, UniTransPose, while not allowing for the same level of direct control over the trade-off, efficiently handles multiple scales by combining (1) an efficient multi-scale transformer encoder that uses both local and global attention with (2) an efficient sub-pixel CNN decoder for better speed and accuracy. Moreover, by incorporating all joints from different benchmarks into a unified skeletal representation, we train robust methods that learn from multiple datasets simultaneously and perform well across a range of scenarios -- including pose variations, lighting conditions, and occlusions. Experiments on six benchmarks demonstrate that the proposed methods significantly outperform state-of-the-art methods while improving computational efficiency. EViTPose exhibits a significant decrease in computational complexity (30% to 44% less in GFLOPs) with a minimal drop of accuracy (0% to 3.5% less), and UniTransPose achieves accuracy improvements ranging from 0.9% to 43.8% across these benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。