用视觉语言大模型将视频转为可理解的动作语义标记,提升识别准确率与可解释性。
Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenization
- 将视频转化为语义一致的动作标记,构建可被大模型理解的行动叙事
- 在NTU RGB+D等数据集上达到94.1%准确率,显著优于传统方法
- 生成自然语言解释,适合需要透明决策过程的应用场景
人类动作识别常面临深层语义理解、复杂上下文信息及细微差异区分的挑战,传统方法在多样化视频数据面前表现受限。受大语言模型能力启发,本文提出LVLM-VAR框架,首次将预训练视觉-语言大模型(LVLMs)应用于视频动作识别,兼顾精度与可解释性。该方法设计视频到语义标记(VST)模块,将原始视频序列转换为离散、语义与时间一致的“动作语义标记”,构建出对LVLM可读的“动作叙事”。这些标记结合自然语言指令,由经LoRA微调的LVLM(如LLaVA-13B)进行动作分类与语义推理。实验表明,LVLM-VAR在NTU RGB+D和NTU RGB+D 120等挑战性基准上达到领先或具有竞争力的表现,例如在NTU RGB+D X-Sub上达94.1%,在NTU RGB+D 120 X-Set上达90.0%,同时显著提升模型可解释性,能生成预测的自然语言解释。
原文摘要 · Abstract (English)
Human action recognition often struggles with deep semantic understanding, complex contextual information, and fine-grained distinction, limitations that traditional methods frequently encounter when dealing with diverse video data. Inspired by the remarkable capabilities of large language models, this paper introduces LVLM-VAR, a novel framework that pioneers the application of pre-trained Vision-Language Large Models (LVLMs) to video action recognition, emphasizing enhanced accuracy and interpretability. Our method features a Video-to-Semantic-Tokens (VST) Module, which innovatively transforms raw video sequences into discrete, semantically and temporally consistent "semantic action tokens," effectively crafting an "action narrative" that is comprehensible to an LVLM. These tokens, combined with natural language instructions, are then processed by a LoRA-fine-tuned LVLM (e.g., LLaVA-13B) for robust action classification and semantic reasoning. LVLM-VAR not only achieves state-of-the-art or highly competitive performance on challenging benchmarks such as NTU RGB+D and NTU RGB+D 120, demonstrating significant improvements (e.g., 94.1% on NTU RGB+D X-Sub and 90.0% on NTU RGB+D 120 X-Set), but also substantially boosts model interpretability by generating natural language explanations for its predictions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。