用文本驱动在线动作检测，零样本也能高效准确。
Text-driven Online Action Detection
- 基于CLIP文本嵌入，实现低开销的视觉语言模型应用
- THUMOS14上达82.46% mAP，超越现有方法
- 适合需要快速适应新动作的实时场景
在线动作检测对视频监控、自动驾驶和人机交互等应用至关重要。该任务需在流式视频中实时分类动作，应对背景噪声与不完整动作。当前主流采用Transformer架构，但近期计算机视觉中视觉语言模型（VLMs）的进展尚未充分应用于该问题，主要受限于高计算成本。本文提出TOAD：一种文本驱动的在线动作检测架构，支持零样本和少样本学习。通过利用CLIP（对比语言-图像预训练）的文本嵌入，实现视觉语言模型的高效应用，无显著计算开销。模型在THUMOS14数据集上达到82.46% mAP，优于现有方法，并在THUMOS14和TVSeries数据集上建立零样本与少样本新基准。
原文摘要 · Abstract (English)
Detecting actions as they occur is essential for applications like video surveillance, autonomous driving, and human-robot interaction. Known as online action detection, this task requires classifying actions in streaming videos, handling background noise, and coping with incomplete actions. Transformer architectures are the current state-of-the-art, yet the potential of recent advancements in computer vision, particularly vision-language models (VLMs), remains largely untapped for this problem, partly due to high computational costs. In this paper, we introduce TOAD: a Text-driven Online Action Detection architecture that supports zero-shot and few-shot learning. TOAD leverages CLIP (Contrastive Language-Image Pretraining) textual embeddings, enabling efficient use of VLMs without significant computational overhead. Our model achieves 82.46% mAP on the THUMOS14 dataset, outperforming existing methods, and sets new baselines for zero-shot and few-shot performance on the THUMOS14 and TVSeries datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。