通过原型引导对齐,提升零样本骨骼动作识别准确率。
Zero-Shot Skeleton-Based Action Recognition With Prototype-Guided Feature Alignment
- 端到端对比学习增强骨骼与文本特征对齐
- 在三个数据集上分别提升22.96%、12.53%、18.54%
- 适合关注跨模态对齐与零样本识别的研究者
零样本骨骼动作识别旨在不依赖训练中未见动作类别的情况下分类未知的骨骼动作,该任务极具挑战性。现有方法通常采用两阶段训练:先在已知动作类别上预训练骨骼编码器,再对齐提取的骨骼与文本特征,依赖骨架-文本对齐和语言模型的泛化能力实现知识迁移。但其效果受限于:1)骨骼特征区分度不足,固定编码器无法捕捉有效对齐信息;2)测试时忽略骨骼与未见文本特征间的对齐偏差。为此,本文提出原型引导特征对齐框架PGFA。我们构建端到端跨模态对比训练机制以强化骨骼-文本对齐,确保骨骼特征充分区分。同时引入原型引导文本特征对齐策略,缓解测试阶段分布差异带来的负面影响。提供理论分析支持该策略,并在NTU-60、NTU-120、PKU-MMD三个数据集上进行实证评估。相比最优基线SMIE方法,本方法在上述数据集上分别提升绝对准确率22.96%、12.53%和18.54%。
原文摘要 · Abstract (English)
Zero-shot skeleton-based action recognition aims to classify unseen skeleton-based human actions without prior exposure to such categories during training. This task is extremely challenging due to the difficulty in generalizing from known to unknown actions. Previous studies typically use two-stage training: pre-training skeleton encoders on seen action categories using cross-entropy loss and then aligning pre-extracted skeleton and text features, enabling knowledge transfer to unseen classes through skeleton-text alignment and language models' generalization. However, their efficacy is hindered by 1) insufficient discrimination for skeleton features, as the fixed skeleton encoder fails to capture necessary alignment information for effective skeleton-text alignment; 2) the neglect of alignment bias between skeleton and unseen text features during testing. To this end, we propose a prototype-guided feature alignment paradigm for zero-shot skeleton-based action recognition, termed PGFA. Specifically, we develop an end-to-end cross-modal contrastive training framework to improve skeleton-text alignment, ensuring sufficient discrimination for skeleton features. Additionally, we introduce a prototype-guided text feature alignment strategy to mitigate the adverse impact of the distribution discrepancy during testing. We provide a theoretical analysis to support our prototype-guided text feature alignment strategy and empirically evaluate our overall PGFA on three well-known datasets. Compared with the top competitor SMIE method, our PGFA achieves absolute accuracy improvements of 22.96%, 12.53%, and 18.54% on the NTU-60, NTU-120, and PKU-MMD datasets, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。