arXiv:2510.13630cs.CV2025-10被引 1

融合音视频信息的轻量级异常检测框架,提升复杂环境下的识别准确率。

AVAR-Net: A Lightweight Audio-Visual Anomaly Recognition Framework with a Benchmark Dataset

  • 用Wav2Vec2和MobileViT提取音视频特征,早期融合后建模长时序依赖
  • 在自建数据集上达89.29%准确率,在XD-Violence上提升2.8%平均精度
  • 适合实际部署,尤其适用于光照差、遮挡等挑战场景

异常识别在监控、交通、医疗和公共安全中至关重要。但现有方法多仅依赖视觉,面对遮挡、低光照和恶劣天气时表现不佳。同时,缺乏大规模同步音视频数据集制约了多模态异常识别发展。为此,本文提出AVAR-Net,一个专为真实环境设计的轻量级音视频异常识别框架。该框架包含音频特征提取器、视频特征提取器、融合策略与序列模式学习网络,能有效建模跨模态关系。其中,Wav2Vec2从原始音频提取鲁棒时序特征,MobileViT捕捉视频帧中的局部与全局视觉表征;采用早期融合机制,并通过多阶段时序卷积网络(MTCN)学习融合表示中的长时序依赖,实现鲁棒的时空推理。此外,本文构建了首个中等规模音视频异常识别基准数据集VAAR,包含3,000段真实世界视频,涵盖10类异常,且音视频同步。实验表明,AVAR-Net在VAAR上达到89.29%准确率,在XD-Violence上取得88.56%平均精度,相比现有最先进方法提升2.8%平均精度,验证了其有效性、高效性与泛化能力,以及VAAR作为基准数据集的实用价值。

原文摘要 · Abstract (English)

Anomaly recognition plays a vital role in surveillance, transportation, healthcare, and public safety. However, most existing approaches rely solely on visual data, making them unreliable under challenging conditions such as occlusion, low illumination, and adverse weather. Moreover, the absence of large-scale synchronized audio-visual datasets has hindered progress in multimodal anomaly recognition. To address these limitations, this study presents AVAR-Net, a lightweight and efficient audio-visual anomaly recognition framework designed for real-world environments. AVAR-Net consists of four main modules: an audio feature extractor, a video feature extractor, fusion strategy, and a sequential pattern learning network that models cross-modal relationships for anomaly recognition. Specifically, the Wav2Vec2 model extracts robust temporal features from raw audio, while MobileViT captures both local and global visual representations from video frames. An early fusion mechanism combines these modalities, and a Multi-Stage Temporal Convolutional Network (MTCN) model that learns long-range temporal dependencies within the fused representation, enabling robust spatiotemporal reasoning. A novel Visual-Audio Anomaly Recognition (VAAR) dataset, is also introduced, serving as a medium-scale benchmark containing 3,000 real-world videos with synchronized audio across ten diverse anomaly classes. Experimental evaluations demonstrate that AVAR-Net achieves 89.29% accuracy on VAAR and 88.56% Average Precision on the XD-Violence dataset, improving Average Precision by 2.8% over existing state-of-the-art methods. These results highlight the effectiveness, efficiency, and generalization capability of the proposed framework, as well as the utility of VAAR as a benchmark for advancing multimodal anomaly recognition research.

异常检测音视频融合轻量化多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。