arXiv:2412.06171cs.CV2024-12CVPR被引 78

构建跨尺度视频异常理解数据集,提升长视频异常检测效率与可解释性。

Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity

  • 通过大模型递归文本标注与人工分割,实现7万+多粒度异常标注
  • 提出自适应采样器,聚焦异常密集帧,显著提升长视频检测效率
  • 适合关注长视频异常分析、可解释性建模的研究者使用

如何让模型理解不同时间尺度和上下文下的视频异常?传统视频异常理解方法仅关注帧级预测,常忽视复杂真实异常的可解释性。近期多模态方法虽融合视觉与文本,却缺乏覆盖短时与长时异常的层次化标注。为此,我们构建了HIVAU-70k,一个大规模跨任意粒度的层次化视频异常理解基准。开发半自动标注引擎,结合人工视频分割与基于大语言模型的递归自由文本标注,生成超过7万条按片段、事件、视频层级组织的多粒度标注。为高效处理长视频异常检测,提出异常导向时间采样器(ATS),将异常评分与密度感知采样结合,自适应选择异常高发帧,使多模态大模型集中于异常区域,显著提升效率与精度。大量实验表明,层级指令数据显著增强异常理解能力,集成ATS与视觉-语言模型优于传统方法。基准与代码已公开于https://github.com/pipixin321/HolmesVAU。

原文摘要 · Abstract (English)

How can we enable models to comprehend video anomalies occurring over varying temporal scales and contexts? Traditional Video Anomaly Understanding (VAU) methods focus on frame-level anomaly prediction, often missing the interpretability of complex and diverse real-world anomalies. Recent multimodal approaches leverage visual and textual data but lack hierarchical annotations that capture both short-term and long-term anomalies. To address this challenge, we introduce HIVAU-70k, a large-scale benchmark for hierarchical video anomaly understanding across any granularity. We develop a semi-automated annotation engine that efficiently scales high-quality annotations by combining manual video segmentation with recursive free-text annotation using large language models (LLMs). This results in over 70,000 multi-granular annotations organized at clip-level, event-level, and video-level segments. For efficient anomaly detection in long videos, we propose the Anomaly-focused Temporal Sampler (ATS). ATS integrates an anomaly scorer with a density-aware sampler to adaptively select frames based on anomaly scores, ensuring that the multimodal LLM concentrates on anomaly-rich regions, which significantly enhances both efficiency and accuracy. Extensive experiments demonstrate that our hierarchical instruction data markedly improves anomaly comprehension. The integrated ATS and visual-language model outperform traditional methods in processing long videos. Our benchmark and model are publicly available at https://github.com/pipixin321/HolmesVAU.

视频异常多模态长视频标注

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。