用视觉事件序列让大模型精准识别长视频中的细微动作。
VT-LVLM-AR: A Video-Temporal Large Vision-Language Model Adapter for Fine-Grained Action Recognition in Long-Term Videos
- 将视频转为语义连贯的视觉事件序列，降低计算负担。
- 在NTU RGB+D数据集上达到94.1%准确率，领先现有方法。
- 适合需要可解释动作识别的长视频分析场景。
长时视频中的人体动作识别因背景复杂、动作细微而对传统深度学习模型构成挑战，主要源于计算开销大、难以捕捉长程时序依赖及语义理解有限。尽管大语言模型（LLMs）和大视觉语言模型（LVLMs）在多模态理解与推理方面表现卓越，但其直接应用于连续视频流进行细粒度动作识别仍属开放问题。本文提出VT-LVLM-AR（视频-时间大视觉语言模型适配器），通过轻量级时空特征提取、自适应时序池化与事件一致性偏置的概念量化，构建视觉事件序列。这些序列输入冻结的LLaVA-1.5模型，结合参数高效提示调优（P-Tuning v2）实现动作分类。在NTU RGB+D与NTU RGB+D 120数据集上的实验证明，该方法持续达到最先进性能（如在NTU RGB+D X-Sub上达94.1%准确率）。消融实验验证了各组件有效性，人工评估也表明视觉事件表示具有高度可解释性。本工作展示了通过有效视频到语言转换与高效模型适配，利用LVLM实现鲁棒且可解释视频动作理解的巨大潜力。
原文摘要 · Abstract (English)
Human action recognition in long-term videos, characterized by complex backgrounds and subtle action differences, poses significant challenges for traditional deep learning models due to computational overhead, difficulty in capturing long-range temporal dependencies, and limited semantic understanding. While Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have shown remarkable capabilities in multi-modal understanding and reasoning, their direct application to continuous video streams for fine-grained action recognition remains an open problem. This paper introduces VT-LVLM-AR (Video-Temporal Large Vision-Language Model Adapter for Action Recognition), a novel framework designed to bridge this gap. VT-LVLM-AR comprises a Video-to-Event Mapper (VTEM) that efficiently transforms raw video into compact, semantically rich, and temporally coherent "visual event sequences" through lightweight spatio-temporal feature extraction, adaptive temporal pooling, and conceptual quantization with an event coherence bias. These visual event sequences are then fed into an LVLM-based Action Reasoning module, specifically a frozen LLaVA-1.5 model, adapted using parameter-efficient Prompt Tuning (P-Tuning v2) for action classification. Comprehensive evaluations on the NTU RGB+D and NTU RGB+D 120 datasets demonstrate that VT-LVLM-AR consistently achieves state-of-the-art performance, surpassing existing methods (e.g., 94.1% accuracy on NTU RGB+D X-Sub). Ablation studies confirm the critical contributions of VTEM's components and the efficacy of Prompt Tuning, while human evaluations underscore the interpretability of our visual event representations. This work highlights the immense potential of leveraging LVLMs for robust and interpretable video action understanding through effective video-to-language translation and efficient model adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。