多模态实时异常检测系统，兼顾工业安全与通用监控。
Multimodal Real-Time Anomaly Detection and Industrial Applications
- 融合视频与音频，用多模型协同实现跨模态感知。
- 在标准硬件上实现实时运行，工业场景准确率显著提升。
- 适合工厂安全、智能安防等需要多源感知的落地应用。
本文设计并实现了完整的多模态房间监测系统，通过同步视频与音频处理，实现活动识别与异常检测。系统迭代了两个版本：初始轻量版采用YOLOv8、ByteTrack和音频频谱变换器（AST）；进阶版引入多模型音频集成（AST、Wav2Vec2、HuBERT）、混合目标检测（YOLO与DETR）、双向跨模态注意力及多方法异常检测。实验表明，该系统在通用监控和工业安全场景中均表现优异，在标准硬件上实现实时性能，兼具高准确率与强鲁棒性。
原文摘要 · Abstract (English)
This paper presents the design, implementation, and evolution of a comprehensive multimodal room-monitoring system that integrates synchronized video and audio processing for real-time activity recognition and anomaly detection. We describe two iterations of the system: an initial lightweight implementation using YOLOv8, ByteTrack, and the Audio Spectrogram Transformer (AST), and an advanced version that incorporates multi-model audio ensembles, hybrid object detection, bidirectional cross-modal attention, and multi-method anomaly detection. The evolution demonstrates significant improvements in accuracy, robustness, and industrial applicability. The advanced system combines three audio models (AST, Wav2Vec2, and HuBERT) for comprehensive audio understanding, dual object detectors (YOLO and DETR) for improved accuracy, and sophisticated fusion mechanisms for enhanced cross-modal learning. Experimental evaluation shows the system's effectiveness in general monitoring scenarios as well as specialized industrial safety applications, achieving real-time performance on standard hardware while maintaining high accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。