arXiv:2504.13460cs.CVcs.AI2025-04中稿 · TIP2026

用多模态推理提升少样本动作定位,融合文本与视觉信息。

Chain-of-Evidence Multimodal Reasoning for Few-shot Temporal Action Localization

  • 设计链式证据推理,结合文本与视频语义对齐
  • 在三个数据集上显著优于现有方法,单/多实例均提升明显
  • 适合关注少样本视频理解与跨模态推理的研究者

传统时间动作定位(TAL)依赖大量精细标注数据,而少样本TAL仅用少量样本即可识别未见动作类别。然而现有方法多仅关注视频级信息,忽略可提供语义支持的文本信息。为此,本文提出基于链式证据(Chain-of-Evidence, CoE)的多模态推理少样本时间动作定位方法。我们设计了一种新框架,通过语义感知的文本-视觉对齐模块,在不同层次对齐查询视频与支持视频。同时,为更好表达文本层面的动作时序依赖与因果关系,提出CoE推理机制,逐步引导视觉语言模型(VLM)与大语言模型(LLM)生成视频的链式证据描述文本。这些文本能捕捉比视觉特征更丰富的动作变化。我们在公开数据集ActivityNet1.3、THUMOS14及新收集的人类异常定位数据集上进行实验,结果表明该方法在单实例和多实例场景下均显著优于现有方法。代码与数据已开源。

原文摘要 · Abstract (English)

Traditional temporal action localization (TAL) methods rely on large amounts of detailed annotated data, whereas few-shot TAL reduces this dependence by using only a few training samples to identify unseen action categories. However, existing few-shot TAL methods typically focus solely on video-level information, neglecting textual information, which can provide valuable semantic support for the action localization task. To address these issues, in this work, we propose a new few-shot temporal action localization method by Chain-of-Evidence multimodal reasoning to improve localization performance. Specifically, we design a novel few-shot learning framework to capture action commonalities and variations, which includes a semantic-aware text-visual alignment module designed to align the query and support videos at different levels. Meanwhile, to better express the temporal dependencies and causal relationships between actions at the textual level, we design a Chain-of-Evidence (CoE) reasoning method that progressively guides the Vision Language Model (VLM) and Large Language Model (LLM) to generate CoE text descriptions for videos. The generated texts can capture more variance of action than visual features. We conduct extensive experiments on the publicly available ActivityNet1.3, THUMOS14 and our newly collected Human-related Anomaly Localization Dataset. The experimental results demonstrate that our proposed method significantly outperforms existing methods in single-instance and multi-instance scenarios. Our source code and data are available at https://github.com/MICLAB-BUPT/VAL-VLM.

少样本学习动作定位多模态推理链式证据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。