arXiv:2510.16989cs.CV2025-10NeurIPS

无需训练即可在线识别视频中执行的步骤,靠大模型零样本能力实现。

Training-free Online Video Step Grounding

  • 用大模型直接判断片段对应步骤,不依赖训练数据
  • 在线推理比传统离线方法更准,且无需任务微调
  • 结合贝叶斯滤波,利用历史帧提升步骤预测稳定性

给定一个任务及其组成步骤,视频步骤定位(VSG)旨在检测视频中执行了哪些步骤。标准方法需要标注的训练集(如步骤级标注或语音描述),收集成本高;且需离线处理整段视频,难以满足在线决策场景。本文探索无需训练、可在线执行的VSG方案。通过利用近期大型多模态模型(LMMs)的零样本能力,我们仅对有限帧序列进行步骤预测,无需完整视频输入。实验表明,这种无训练在线策略优于离线训练模型。受此启发,提出贝叶斯定位框架BaGLM,将历史帧信息融入LMM预测:利用大语言模型提取步骤转移依赖矩阵,并估计步骤进展程度。在三个数据集上的实验显示,BaGLM性能超越现有训练型离线方法。

原文摘要 · Abstract (English)

Given a task and a set of steps composing it, Video Step Grounding (VSG) aims to detect which steps are performed in a video. Standard approaches for this task require a labeled training set (e.g., with step-level annotations or narrations), which may be costly to collect. Moreover, they process the full video offline, limiting their applications for scenarios requiring online decisions. Thus, in this work, we explore how to perform VSG online and without training. We achieve this by exploiting the zero-shot capabilities of recent Large Multimodal Models (LMMs). In particular, we use LMMs to predict the step associated with a restricted set of frames, without access to the whole video. We show that this online strategy without task-specific tuning outperforms offline and training-based models. Motivated by this finding, we develop Bayesian Grounding with Large Multimodal Models (BaGLM), further injecting knowledge of past frames into the LMM-based predictions. BaGLM exploits Bayesian filtering principles, modeling step transitions via (i) a dependency matrix extracted through large language models and (ii) an estimation of step progress. Experiments on three datasets show superior performance of BaGLM over state-of-the-art training-based offline methods.

视频定位零样本在线推理大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。