轻量级多模态框架CM3T，高效适配不同数据的视频分类任务。
CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets
- 引入双适配器结构：视觉适配器与跨注意力适配器，实现多模态迁移。
- 仅需12.8%参数训练视频输入，22.3%参数处理额外模态，性能媲美甚至超越现有方法。
- 无需预训练重训练，适合资源有限的实际视频应用部署。
跨学习面临数据异质性或不足、大模型重训练资源消耗大的挑战。受NLP中迁移学习启发，本文提出一种新型模型无关插件架构CM3T，用于将Transformer模型适配至新信息。设计两种适配器模块：多头视觉适配器用于迁移学习，跨注意力适配器用于多模态学习。训练效率显著提升，因骨干网络及其他插件无需微调。在Epic-Kitchens-100、MPIIGroupInteraction和UDIVA v0.5三个数据集上进行对比与消融实验，验证了该框架在不同录制条件与任务下的有效性。处理视频输入仅需12.8%可训练参数，新增两模态仅需22.3%可训练参数，性能达到甚至超过当前最优水平。CM3T对训练或预训练无特殊要求，是推动通用模型向具体视频分类应用落地的重要一步。
原文摘要 · Abstract (English)
Challenges in cross-learning involve inhomogeneous or even inadequate amount of training data and lack of resources for retraining large pretrained models. Inspired by transfer learning techniques in NLP, adapters and prefix tuning, this paper presents a new model-agnostic plugin architecture for cross-learning, called CM3T, that adapts transformer-based models to new or missing information. We introduce two adapter blocks: multi-head vision adapters for transfer learning and cross-attention adapters for multimodal learning. Training becomes substantially efficient as the backbone and other plugins do not need to be finetuned along with these additions. Comparative and ablation studies on three datasets Epic-Kitchens-100, MPIIGroupInteraction and UDIVA v0.5 show efficacy of this framework on different recording settings and tasks. With only 12.8% trainable parameters compared to the backbone to process video input and only 22.3% trainable parameters for two additional modalities, we achieve comparable and even better results than the state-of-the-art. CM3T has no specific requirements for training or pretraining and is a step towards bridging the gap between a general model and specific practical applications of video classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。