用文本代替视频训练异常检测模型,突破数据依赖难题
TD-VAD: Breaking Visual Dependence in Video Anomaly Detection with Text-Driven Learning

- 用大模型生成带时序特性的文本描述,替代视频数据训练
- 在两个大规模数据集上显著超越现有单类与无监督方法
- 适合缺乏标注异常视频的场景,尤其适用于罕见事件检测
现有视频异常检测(VAD)方法通常依赖视觉数据训练,但异常样本稀少且类型多样,难以获取足够标注数据。本文提出文本驱动的视频异常检测(TD-VAD),首次将文本视为视频序列进行建模,利用大语言模型生成具有时序特征的文本描述,无需目标域异常视频数据即可训练模型。为捕捉事件的长短时序逻辑,设计事件演化因果注意力模块以建模跨时间上下文依赖。推理时,通过冻结的CLIP编码器提取视频帧嵌入,对齐文本与视频模态,同时保留关键视觉信息。在XD-Violence和UCF-Crime两个大规模数据集上的实验表明,该方法显著优于现有的单类与无监督方法。
原文摘要 · Abstract (English)
Visual data is typically a prerequisite for training existing video anomaly detection (VAD) methods. However, obtaining sufficient annotated anomaly data for training is challenging and not scalable due to the rarity of anomaly data and the wide variety of abnormal events. In this work, we advocate that the effectiveness of treating texts as video sequences for the VAD model and propose a novel Text-Driven Video Anomaly Detection (TD-VAD) approach to break visual dependence. In contrast to the anomaly video data, text descriptions of abnormal events are easy to collect, and their class labels can be directly derived. Specifically, our method utilizes video-like text descriptions with temporal characteristics generated by LLM to train a VAD model, without any reliance on target-domain anomaly data. To capture the long- and short-range temporal logic of events, we design the event evolution causal attention module to model contextual dependencies across time. During inference, considering the domain gap between the texts and video sequences, we use the frozen CLIP encoder to extract embeddings of video frames to align the text modality while retaining crucial visual information. Comprehensive experiments on two large-scale VAD datasets, XD-Violence and UCF-Crime, demonstrate that our method outperforms prior one-class and unsupervised VAD methods by a large margin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。