用大模型自信度实现零样本动作定位,无需训练即可精准识别长视频中的动作起止
Zero-shot Action Localization via the Confidence of Large Vision-Language Models
- 利用大语言模型生成动作起止的详细描述作为查询
- 通过视觉语言模型生成帧级置信度,聚合后定位动作
- 无需训练,在复杂数据集上表现优异,适合医疗和体育场景
在专业体育和微创手术等领域,精确识别未剪辑视频中的特定动作对分析至关重要。然而,许多场景缺乏大规模带标注的视频数据,限制了视频理解模型的微调。近年来,大视觉语言模型(LVLM)在多种视频理解任务中展现出强大的零样本能力。但将具备强大视觉问答能力的LVLM应用于长视频的零样本动作定位仍鲜有研究。为此,我们提出真正的零样本动作定位方法(ZEAL)。具体地,利用大语言模型(LLM)内置的动作知识,将动作扩展为典型起止时刻的详细描述,作为查询输入LVLM,生成帧级置信度分数并聚合得到定位结果。该方法简单灵活,可适配未来更强大的LVLM。我们在一个具有挑战性的基准上实现了卓越的零样本定位性能,且无需任何训练。代码已公开:github.com/josaklil-ai/zeal。
原文摘要 · Abstract (English)
Precise action localization in untrimmed video is vital for fields such as professional sports and minimally invasive surgery, where the delineation of particular motions in recordings can dramatically enhance analysis. But in many cases, large scale datasets with video-label pairs for localization are unavailable, limiting the opportunity to fine-tune video-understanding models. Recent developments in large vision-language models (LVLM) address this need with impressive zero-shot capabilities in a variety of video understanding tasks. However, the adaptation of LVLMs, with their powerful visual question answering capabilities, to zero-shot localization in long-form video is still relatively unexplored. To this end, we introduce a true Zero-shot Action Localization method (ZEAL). Specifically, we leverage the built-in action knowledge of a large language model (LLM) to inflate actions into detailed descriptions of the archetypal start and end of the action. These descriptions serve as queries to LVLM for generating frame-level confidence scores which can be aggregated to produce localization outputs. The simplicity and flexibility of our method lends it amenable to more capable LVLMs as they are developed, and we demonstrate remarkable results in zero-shot action localization on a challenging benchmark, without any training. Our code is publicly available at $\href{https://github.com/josaklil-ai/zeal}{github.com/josaklil-ai/zeal}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。