arXiv:2505.06002cs.CV2025-05被引 1

通过任务自适应与顺序对齐,提升少样本动作识别的准确率。

Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognition

  • 为图像编码器设计任务特定适配器,增强特征判别性。
  • 用大语言模型生成动作子阶段描述,建模语义顺序关系。
  • 细粒度跨模态对齐,使视觉特征与语义时间阶段对齐。

大规模预训练模型在语言和图像任务中表现优异,促使研究者探索将其应用于少样本动作识别(FSAR)。然而现有方法存在四大问题:直接微调会削弱模型泛化能力;视觉任务中任务特异性信息挖掘不足;文本建模常忽略语义顺序;跨模态对齐未考虑多模态的时间耦合。为此,我们提出Task-Adapter++,一种参数高效的双模态适配方法。针对不同任务的差异,设计图像编码器的任务特定适配器,使特征提取更关注判别性信息。利用大语言模型(LLMs)为每类动作生成详细的子动作序列描述,并在文本编码器中引入语义顺序适配器,有效建模子动作间的时序关系。进一步提出细粒度跨模态对齐策略,主动将视觉特征映射至与语义描述相同的时间阶段。在5个基准数据集上实验表明,该方法持续达到当前最优性能。代码已开源:https://github.com/Jaulin-Bage/Task-Adapter-pp。

原文摘要 · Abstract (English)

Large-scale pre-trained models have achieved remarkable success in language and image tasks, leading an increasing number of studies to explore the application of pre-trained image models, such as CLIP, in the domain of few-shot action recognition (FSAR). However, current methods generally suffer from several problems: 1) Direct fine-tuning often undermines the generalization capability of the pre-trained model; 2) The exploration of task-specific information is insufficient in the visual tasks; 3) The semantic order information is typically overlooked during text modeling; 4) Existing cross-modal alignment techniques ignore the temporal coupling of multimodal information. To address these, we propose Task-Adapter++, a parameter-efficient dual adaptation method for both image and text encoders. Specifically, to make full use of the variations across different few-shot learning tasks, we design a task-specific adaptation for the image encoder so that the most discriminative information can be well noticed during feature extraction. Furthermore, we leverage large language models (LLMs) to generate detailed sequential sub-action descriptions for each action class, and introduce semantic order adapters into the text encoder to effectively model the sequential relationships between these sub-actions. Finally, we develop an innovative fine-grained cross-modal alignment strategy that actively maps visual features to reside in the same temporal stage as semantic descriptions. Extensive experiments fully demonstrate the effectiveness and superiority of the proposed method, which achieves state-of-the-art performance on 5 benchmarks consistently. The code is open-sourced at https://github.com/Jaulin-Bage/Task-Adapter-pp.

少样本学习动作识别跨模态对齐顺序建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。