arXiv:2605.13202cs.CVcs.AI2026-05中稿 · publication in IEE…

解决少样本动作识别中语义与时间对齐难题,提升模型泛化能力。

STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

论文配图:STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition
图 1 · 摘自论文原文
  • 引入时序语义注意力机制,实现帧级跨模态对齐。
  • 在1-shot设置下,显著提升SSv2和HMDB51的准确率,最高增益8.1%。
  • 适合需要小样本高效学习的动作识别研究者使用。

少样本动作识别(FSAR)要求模型仅凭少量标注样本就能泛化到新动作类别。尽管视觉-语言模型取得进展,现有方法仍存在语义-时间错位问题:静态文本提示无法捕捉序列中稀疏出现的关键视觉线索;且多尺度时序建模不足,短期判别性特征与长程依赖常被过度平滑或碎片化。为此,我们提出统一框架STAR,包含语义对齐与时序感知模块,有效弥合语义-时间鸿沟,并将Mamba的序列建模能力迁移至FSAR。语义对齐模块设计时序语义注意力(TSA),实现帧级跨模态对齐,确保细粒度语义-时间一致性。时序感知模块融合语义引导的Mamba块、多频时序采样与双向状态空间精炼,生成具有增强判别力与时序一致性的语义对齐原型。此外,基于大语言模型生成的时序依赖类别描述提供长程语义指导。在五个FSAR基准上大量实验表明,STAR持续优于现有最先进方法。例如,在1-shot设置下,于SSv2-Full与SSv2-Small数据集上分别提升8.1%与6.7%,在HMDB51上提升7.3%,验证了其在有限监督下的有效性。代码已公开于https://github.com/HongliLiu1/STAR-main。

原文摘要 · Abstract (English)

Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples. Despite progress with vision-language models, existing approaches still suffer from semantic-temporal misalignment, where static textual prompts fail to capture decisive visual cues that appear sparsely across sequences, and from inadequate modeling of multi-scale temporal dynamics, as short-term discriminative cues and long-range dependencies are often either oversmoothed or fragmented. To address these challenges, we propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework, consisting of a semantic-alignment component and a temporal-aware component, effectively bridging the semantic and temporal gaps and transferring the sequence modeling capability of Mamba into the FSAR. The semantic alignment module introduces a Temporal Semantic Attention (TSA) mechanism, which performs frame-level cross-modal alignment with textual cues, ensuring fine-grained semantic-temporal consistency. The temporal-aware module incorporates a Semantic Temporal Prototype Refiner (STPR) that integrates semantic-guided Mamba blocks with multi-frequency temporal sampling and bidirectional state-space refinement, yielding semantically aligned prototypes with enhanced discriminative fidelity and temporal consistency. Furthermore, temporally dependent class descriptors derived from large language models (LLMs) provide long-range semantic guidance. Extensive experiments on five FSAR benchmarks demonstrate the consistent superiority of STAR over state-of-the-art methods. For instance, STAR achieves up to 8.1% and 6.7% gains on the SSv2-Full and SSv2-Small datasets under the 1-shot setting, and 7.3% on HMDB51, validating its effectiveness under limited supervision. The code is available at https://github.com/HongliLiu1/STAR-main.

少样本识别时序建模跨模态对齐Mamba

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。