arXiv:2411.11288cs.CV2024-11CVPR被引 21

提出神经元式动态演化框架,提升零样本骨骼动作识别泛化能力

Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition

  • 构建时空演化的微原型,融合上下文感知侧信息逐步对齐骨架与语义
  • 通过空间压缩与时间记忆机制,精准捕捉结构与规律性特征
  • 类比神经元生长过程,有效支持未见动作类别泛化,适合零样本学习研究者

零样本骨骼动作识别需依赖已见类别和共享语义实现鲁棒的未知泛化。现有方法常采用不可控映射与显著表征,难以捕捉精细跨模态关联。为此,我们提出基于上下文感知侧信息引导的动态演化双模态协同框架Neuron,从时空层面微观到宏观探索更细粒度的跨模态对应关系。具体地,1)构建时空演化微原型,结合动态上下文感知侧信息,逐步捕获骨架-语义间复杂协同关系,持续优化跨模型对齐;2)引入空间压缩与时间记忆机制,指导微原型成长,使其吸收结构相关的空间表征与依赖规律的时间模式。该过程类比神经元学习与生长,赋予框架泛化至新未见动作类别的能力。在多个基准数据集上的实验验证了方法优势。

原文摘要 · Abstract (English)

Zero-shot skeleton action recognition is a non-trivial task that requires robust unseen generalization with prior knowledge from only seen classes and shared semantics. Existing methods typically build the skeleton-semantics interactions by uncontrollable mappings and conspicuous representations, thereby can hardly capture the intricate and fine-grained relationship for effective cross-modal transferability. To address these issues, we propose a novel dyNamically Evolving dUal skeleton-semantic syneRgistic framework with the guidance of cOntext-aware side informatioN (dubbed Neuron), to explore more fine-grained cross-modal correspondence from micro to macro perspectives at both spatial and temporal levels, respectively. Concretely, 1) we first construct the spatial-temporal evolving micro-prototypes and integrate dynamic context-aware side information to capture the intricate and synergistic skeleton-semantic correlations step-by-step, progressively refining cross-model alignment; and 2) we introduce the spatial compression and temporal memory mechanisms to guide the growth of spatial-temporal micro-prototypes, enabling them to absorb structure-related spatial representations and regularity-dependent temporal patterns. Notably, such processes are analogous to the learning and growth of neurons, equipping the framework with the capacity to generalize to novel unseen action categories. Extensive experiments on various benchmark datasets demonstrated the superiority of the proposed method.

零样本识别骨骼动作动态演化跨模态对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。