高效处理流式视频异常检测,支持动态定义与实时推理。
ESOM: Efficiently Understanding Streaming Video Anomalies with Open-world Dynamic Definitions
- 无需训练,通过提示规范化和帧间令牌压缩提升效率
- 单卡实现实时处理,时间定位准确率显著优于现有方法
- 适合智能监控与直播内容审核等动态场景应用
开放世界视频异常检测(OWVAD)旨在应对不同异常定义下的异常事件识别与解释,对智能监控和直播内容审核至关重要。现有基于多模态大模型的方法虽具较强泛化能力,但仍存在部署效率低、不支持流式处理、难以适应动态异常定义等问题。本文提出ESOM,一种无需训练的高效流式OWVAD模型。其包含:定义归一化模块以减少幻觉;帧间匹配的帧内令牌合并模块压缩冗余视觉特征;混合流式记忆模块实现高效因果推理;概率评分模块将区间级文本输出转为帧级异常分数。此外,本文构建了OpenDef-Bench基准,包含清洁监控视频与多样自然异常定义,用于评估不同条件下的性能。大量实验表明,ESOM在单张GPU上实现实时推理,并在异常时间定位、分类及描述生成任务中达到当前最优表现。代码与基准已开源。
原文摘要 · Abstract (English)
Open-world video anomaly detection (OWVAD) aims to detect and explain abnormal events under different anomaly definitions, which is important for applications such as intelligent surveillance and live-streaming content moderation. Recent MLLM-based methods have shown promising open-world generalization, but still suffer from three major limitations: inefficiency for practical deployment, lack of streaming processing adaptation, and limited support for dynamic anomaly definitions in both modeling and evaluation. To address these issues, this paper proposes ESOM, an efficient streaming OWVAD model that operates in a training-free manner. ESOM includes a Definition Normalization module to structure user prompts for reducing hallucination, an Inter-frame-matched Intra-frame Token Merging module to compress redundant visual tokens, a Hybrid Streaming Memory module for efficient causal inference, and a Probabilistic Scoring module that converts interval-level textual outputs into frame-level anomaly scores. In addition, this paper introduces OpenDef-Bench, a new benchmark with clean surveillance videos and diverse natural anomaly definitions for evaluating performance under varying conditions. Extensive experiments show that ESOM achieves real-time efficiency on a single GPU and state-of-the-art performance in anomaly temporal localization, classification, and description generation. The code and benchmark will be released at https://github.com/Kamino666/ESOM_OpenDef-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。