用提示词调节语义原型,让模型更好区分视觉相似动作。
Distinguishing Visually Similar Actions: Prompt-Guided Semantic Prototype Modulation for Few-Shot Action Recognition
- 通过分层运动优化减少背景干扰,强化动态特征捕捉。
- 用文本提示调制视觉原型,提升相似动作的区分能力。
- 适合需要快速学习新动作的少样本场景,如视频分析应用。
少样本动作识别旨在使模型仅凭少量标注样本即可快速学习新动作类别,以应对现实应用中的数据稀缺问题。当前研究面临三大核心挑战:(1)时间建模中模型易受无关静态背景干扰,难以捕捉动态动作本质;(2)视觉相似性导致细微差异动作难以区分;(3)视觉-文本支持原型与纯视觉查询间存在模态鸿沟,影响共享嵌入空间中的对齐。为此,本文提出CLIP-SPM框架,包含三个组件:(1)分层协同运动精炼(HSMR)模块,对齐深层与浅层运动特征,降低背景干扰,增强时序建模;(2)语义原型调制(SPM)策略,生成与查询相关的文本提示,融合视觉特征,提升相似动作的判别力;(3)原型-锚点双调制(PADM)方法,优化支持原型并引导查询特征对齐全局语义锚点,增强支持与查询样本的一致性。在Kinetics、SSv2-Full、SSv2-Small、UCF101和HMDB51等标准基准上的实验表明,该方法在1-shot、3-shot和5-shot设置下均取得具有竞争力的性能。大量消融实验与可视化分析验证了各组件的有效性及其对核心挑战的贡献。源代码与模型已公开于GitHub。
原文摘要 · Abstract (English)
Few-shot action recognition aims to enable models to quickly learn new action categories from limited labeled samples, addressing the challenge of data scarcity in real-world applications. Current research primarily addresses three core challenges: (1) temporal modeling, where models are prone to interference from irrelevant static background information and struggle to capture the essence of dynamic action features; (2) visual similarity, where categories with subtle visual differences are difficult to distinguish; and (3) the modality gap between visual-textual support prototypes and visual-only queries, which complicates alignment within a shared embedding space. To address these challenges, this paper proposes a CLIP-SPM framework, which includes three components: (1) the Hierarchical Synergistic Motion Refinement (HSMR) module, which aligns deep and shallow motion features to improve temporal modeling by reducing static background interference; (2) the Semantic Prototype Modulation (SPM) strategy, which generates query-relevant text prompts to bridge the modality gap and integrates them with visual features, enhancing the discriminability between similar actions; and (3) the Prototype-Anchor Dual Modulation (PADM) method, which refines support prototypes and aligns query features with a global semantic anchor, improving consistency across support and query samples. Comprehensive experiments across standard benchmarks, including Kinetics, SSv2-Full, SSv2-Small, UCF101, and HMDB51, demonstrate that our CLIP-SPM achieves competitive performance under 1-shot, 3-shot, and 5-shot settings. Extensive ablation studies and visual analyses further validate the effectiveness of each component and its contributions to addressing the core challenges. The source code and models are publicly available at GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。