用2D轮廓重建3D特征,提升步态识别精度与效率
Mesh-Gait: A Unified Framework for Gait Recognition Through Multi-Modal Representation Learning from 2D Silhouettes
- 通过2D轮廓重建3D热图作为中间表示,融合多模态信息
- 在CASIA-B数据集上达到98.7%准确率,超越现有方法
- 计算高效,适合实时步态识别,适用于安防与监控场景
步态识别是一种基于独特行走模式的生物特征技术,通常使用2D轮廓或骨骼表示。然而,这些方法在视角变化、遮挡和噪声下表现不佳。结合3D体形信息的多模态方法虽更鲁棒,但计算开销大,难以实现实时应用。为此,我们提出Mesh-Gait,一种端到端的多模态步态识别框架,可直接从2D轮廓重建3D表示,有效结合两种模态优势。为避免直接学习3D关节或网格带来的复杂性,Mesh-Gait采用3D热图为中间表示,使模型能有效捕捉3D几何信息,同时保持简洁与高效。训练中,3D热图逐步重构并趋于精确,损失函数基于重建的3D关节、虚拟标记与3D网格与真实值的差异,确保空间对齐与结构一致性。模型从轮廓和重建的3D热图中高效提取判别性特征,避免直接从RGB视频重建3D的高开销,使网络聚焦于运动动态而非无关视觉细节。大量实验表明,Mesh-Gait在性能上达到当前最优,在CASIA-B数据集上准确率达98.7%。代码将在论文接收后公开。
原文摘要 · Abstract (English)
Gait recognition, a fundamental biometric technology, leverages unique walking patterns for individual identification, typically using 2D representations such as silhouettes or skeletons. However, these methods often struggle with viewpoint variations, occlusions, and noise. Multi-modal approaches that incorporate 3D body shape information offer improved robustness but are computationally expensive, limiting their feasibility for real-time applications. To address these challenges, we introduce Mesh-Gait, a novel end-to-end multi-modal gait recognition framework that directly reconstructs 3D representations from 2D silhouettes, effectively combining the strengths of both modalities. Compared to existing methods, directly learning 3D features from 3D joints or meshes is complex and difficult to fuse with silhouette-based gait features. To overcome this, Mesh-Gait reconstructs 3D heatmaps as an intermediate representation, enabling the model to effectively capture 3D geometric information while maintaining simplicity and computational efficiency. During training, the intermediate 3D heatmaps are gradually reconstructed and become increasingly accurate under supervised learning, where the loss is calculated between the reconstructed 3D joints, virtual markers, and 3D meshes and their corresponding ground truth, ensuring precise spatial alignment and consistent 3D structure. Mesh-Gait extracts discriminative features from both silhouettes and reconstructed 3D heatmaps in a computationally efficient manner. This design enables the model to capture spatial and structural gait characteristics while avoiding the heavy overhead of direct 3D reconstruction from RGB videos, allowing the network to focus on motion dynamics rather than irrelevant visual details. Extensive experiments demonstrate that Mesh-Gait achieves state-of-the-art accuracy. The code will be released upon acceptance of the paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。