arXiv:2411.19041cs.CV2024-11CVPR被引 8

提出TAMT方法,高效解决跨域少样本动作识别难题

TAMT: Temporal-Aware Model Tuning for Cross-Domain Few-Shot Action Recognition

  • 分离预训练与微调流程,避免重复训练
  • 通过时序自适应模块提升模型迁移性能,最高提升31%
  • 适合资源受限下快速适配新动作数据集的场景

超越传统少样本动作识别(FSAR),跨域少样本动作识别(CDFSAR)因需解决源域到目标域的域差异而受到关注。现有方法多采用源域与目标域联合训练以缓解域偏移问题,但存在两大局限:一是成对联合训练需为每个目标域重训练模型,计算开销大,尤其在源域大、目标域小时更明显;二是联合训练后的预训练模型直接用于目标域,未能充分挖掘其潜力,限制识别性能。为此,本文提出一种简单而有效的基线方法——时序感知模型微调(TAMT)。TAMT采用解耦范式,先在源域上进行预训练,再对目标域进行微调,避免多个目标域共享单一源域时的重复训练。为高效探索预训练模型在目标域中的潜力,TAMT引入分层时序微调网络(HTTN),核心包含局部时序自适应适配器(TAA)和全局时序自适应时刻微调(GTMT)。其中,TAA仅学习少量参数,重新校准冻结预训练模型的中间特征,实现高效领域适配;GTMT则生成强大视频表征,提升目标域匹配性能。在多个主流视频基准上的实验表明,TAMT相比近期先进方法提升13%~31%,达到新的最佳性能。

原文摘要 · Abstract (English)

Going beyond few-shot action recognition (FSAR), cross-domain FSAR (CDFSAR) has attracted recent research interests by solving the domain gap lying in source-to-target transfer learning. Existing CDFSAR methods mainly focus on joint training of source and target data to mitigate the side effect of domain gap. However, such kind of methods suffer from two limitations: First, pair-wise joint training requires retraining deep models in case of one source data and multiple target ones, which incurs heavy computation cost, especially for large source and small target data. Second, pre-trained models after joint training are adopted to target domain in a straightforward manner, hardly taking full potential of pre-trained models and then limiting recognition performance. To overcome above limitations, this paper proposes a simple yet effective baseline, namely Temporal-Aware Model Tuning (TAMT) for CDFSAR. Specifically, our TAMT involves a decoupled paradigm by performing pre-training on source data and fine-tuning target data, which avoids retraining for multiple target data with single source. To effectively and efficiently explore the potential of pre-trained models in transferring to target domain, our TAMT proposes a Hierarchical Temporal Tuning Network (HTTN), whose core involves local temporal-aware adapters (TAA) and a global temporal-aware moment tuning (GTMT). Particularly, TAA learns few parameters to recalibrate the intermediate features of frozen pre-trained models, enabling efficient adaptation to target domains. Furthermore, GTMT helps to generate powerful video representations, improving match performance on the target domain. Experiments on several widely used video benchmarks show our TAMT outperforms the recently proposed counterparts by 13%$\sim$31%, achieving new state-of-the-art CDFSAR results.

跨域识别少样本学习时序建模模型微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。