用极少参数提升视频语言模型的迁移能力,避免遗忘旧知识。
Efficient Transfer Learning for Video-language Foundation Models

- 设计轻量级多模态时空适配器,仅用2-7%参数增强图文对齐。
- 引入时空描述一致性约束,减少过拟合,提升模型泛化性。
- 适用于零样本、小样本等各类迁移任务,适合资源受限场景。
预训练视觉语言模型为下游任务提供了强大的迁移学习基础。在视频动作识别领域,主流方法通常引入额外模块以捕捉时序信息,但这些模块显著增加参数量,并易导致原有通用知识的灾难性遗忘。本文提出一种参数高效的多模态时空适配器(MSTA),增强文本与视觉表征间的对齐,平衡通用知识与任务特异性适应。为缓解过拟合并提升泛化能力,我们引入时空描述引导的一致性约束:将模板输入(如“a video of {cls}”)送入可训练语言分支,同时将大模型生成的时空描述送入预训练语言分支,强制两分支输出一致。该机制有效降低对下游任务的过拟合,增强可训练分支在时空语义空间中的区分性。我们在四个任务上评估:零样本迁移、少样本学习、基类到新类泛化、全监督学习。相比多种先进方法,本方法在所有评估中表现优异,且仅使用原模型2-7%的可训练参数。
原文摘要 · Abstract (English)
Pre-trained vision-language models provide a robust foundation for efficient transfer learning across various downstream tasks. In the field of video action recognition, mainstream approaches often introduce additional modules to capture temporal information. Although the additional modules increase the capacity of model, enabling it to better capture video-specific inductive biases, existing methods typically introduce a substantial number of new parameters and are prone to catastrophic forgetting of previously acquired generalizable knowledge. In this paper, we propose a parameter-efficient Multi-modal Spatio-Temporal Adapter (MSTA) to enhance the alignment between textual and visual representations, achieving a balance between generalizable knowledge and task-specific adaptation. Furthermore, to mitigate over-fitting and enhance generalizability, we introduce a spatio-temporal description-guided consistency constraint.This constraint involves providing template inputs (e.g., "a video of \{\textbf{cls}\}") to the trainable language branch and LLM-generated spatio-temporal descriptions to the pre-trained language branch, enforcing output consistency between the branches. This approach reduces overfitting to downstream tasks and enhances the distinguishability of the trainable branch within the spatio-temporal semantic space. We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-supervised learning. Compared to many state-of-the-art methods, our MSTA achieves outstanding performance across all evaluations, while using only 2-7\% of the trainable parameters in the original model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。