通过双模态对齐实现骨架动作零样本识别,提升模型泛化能力。
Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignment
- 设计双对齐模块:直接映射与增强对齐,融合视觉与文本特征
- 在多个基准上达到最新性能,显著缩小跨模态差距
- 适合研究零样本学习与动作识别的学者使用
零样本动作识别旨在解决动作识别中的可扩展性与泛化问题,使模型能动态适应新出现的未见动作,是计算机视觉的重要研究方向。其核心在于对齐视觉特征与表征动作类别的语义向量。现有方法或直接将视觉特征投影至文本语义空间,或学习两模态间的共享嵌入空间,但直接投影难以精准对齐,且学习鲁棒、判别性强的嵌入空间常具挑战。为此,本文提出针对骨架数据的双视觉-文本对齐方法(DVTA),包含两个对齐模块:直接对齐(DA)与增强对齐(AA),以及语义描述增强(SDE)。DA模块通过专用视觉投影器将骨架特征映射至语义空间,并经基于交叉注意力的SDE增强骨架与文本的关联,缩小模态差异;AA模块利用深度度量学习强化骨架与文本间的相似性建模。本方法在多个主流零样本骨架动作识别基准上取得当前最优性能。代码已公开于:https://github.com/jidongkuang/DVTA。
原文摘要 · Abstract (English)
Zero-shot action recognition, which addresses the issue of scalability and generalization in action recognition and allows the models to adapt to new and unseen actions dynamically, is an important research topic in computer vision communities. The key to zero-shot action recognition lies in aligning visual features with semantic vectors representing action categories. Most existing methods either directly project visual features onto the semantic space of text category or learn a shared embedding space between the two modalities. However, a direct projection cannot accurately align the two modalities, and learning robust and discriminative embedding space between visual and text representations is often difficult. To address these issues, we introduce Dual Visual-Text Alignment (DVTA) for skeleton-based zero-shot action recognition. The DVTA consists of two alignment modules--Direct Alignment (DA) and Augmented Alignment (AA)--along with a designed Semantic Description Enhancement (SDE). The DA module maps the skeleton features to the semantic space through a specially designed visual projector, followed by the SDE, which is based on cross-attention to enhance the connection between skeleton and text, thereby reducing the gap between modalities. The AA module further strengthens the learning of the embedding space by utilizing deep metric learning to learn the similarity between skeleton and text. Our approach achieves state-of-the-art performances on several popular zero-shot skeleton-based action recognition benchmarks. The code is available at: https://github.com/jidongkuang/DVTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。