用多模态大模型引导弱监督动作定位，提升定位准确率
Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models
- 利用多模态大模型提供动作语义和完整语义先验
- 设计两个模块协同解决定位不全或过度问题
- 适用于各类弱监督动作定位模型，提升泛化能力
多模态大语言模型（MLLM）在视频理解中表现优异，能融合视频基础模型与语言模型，突破预设视觉任务的限制。然而其高算力需求仍使传统模型具重要地位。本文提出新型学习范式MLLM4WTAL，利用MLLM提供时序动作的关键语义与完整语义先验，指导传统弱监督动作定位（WTAL）方法。通过引入关键语义匹配（KSM）与完整语义重建（CSR）双模块协同机制，有效缓解WTAL中常见的定位不全与过泛问题。大量实验证明，该方法可显著提升多种异构WTAL模型的性能。
原文摘要 · Abstract (English)
Recent breakthroughs in Multimodal Large Language Models (MLLMs) have gained significant recognition within the deep learning community, where the fusion of the Video Foundation Models (VFMs) and Large Language Models(LLMs) has proven instrumental in constructing robust video understanding systems, effectively surmounting constraints associated with predefined visual tasks. These sophisticated MLLMs exhibit remarkable proficiency in comprehending videos, swiftly attaining unprecedented performance levels across diverse benchmarks. However, their operation demands substantial memory and computational resources, underscoring the continued importance of traditional models in video comprehension tasks. In this paper, we introduce a novel learning paradigm termed MLLM4WTAL. This paradigm harnesses the potential of MLLM to offer temporal action key semantics and complete semantic priors for conventional Weakly-supervised Temporal Action Localization (WTAL) methods. MLLM4WTAL facilitates the enhancement of WTAL by leveraging MLLM guidance. It achieves this by integrating two distinct modules: Key Semantic Matching (KSM) and Complete Semantic Reconstruction (CSR). These modules work in tandem to effectively address prevalent issues like incomplete and over-complete outcomes common in WTAL methods. Rigorous experiments are conducted to validate the efficacy of our proposed approach in augmenting the performance of various heterogeneous WTAL models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。