用多模态大模型让交通视频自动分析事故,对话式交互更高效。
When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis
- 引入多模态大模型代理,实现视频分析从提取到对话的转变。
- 在丰田数据集上准确完成事故分类与视觉定位,支持长时视频处理。
- 适合交通管理、智能驾驶研发人员快速分析海量监控视频。
24/7/365全天候运行的交通监控视频具备提升事故时空覆盖的巨大潜力,但对成百上千路摄像头的持续分析仍是难题。现有视觉方法多聚焦于车辆轨迹等原始信息提取,需大量人工后处理才能获得可操作洞察。本文提出SeeUnsafe框架,利用多模态大语言模型(MLLM)代理,将传统的“提取-解释”流程升级为交互式对话模式。该方法通过基于严重性的聚合策略处理不同长度视频,并设计新型多模态提示生成结构化响应,支持细粒度视觉定位。我们引入IMS(信息匹配得分)作为基于MLLM的新评估指标,用于对齐结构化输出与真实标签。在Toyota Woven Traffic Safety数据集上的实验表明,借助现成的MLLM,SeeUnsafe能有效实现事故感知的视频分类与视觉定位。源代码将公开于\url{https://github.com/ai4ce/SeeUnsafe}。
原文摘要 · Abstract (English)
The increasing availability of traffic videos functioning on a 24/7/365 time scale has the great potential of increasing the spatio-temporal coverage of traffic accidents, which will help improve traffic safety. However, analyzing footage from hundreds, if not thousands, of traffic cameras in a 24/7/365 working protocol remains an extremely challenging task, as current vision-based approaches primarily focus on extracting raw information, such as vehicle trajectories or individual object detection, but require laborious post-processing to derive actionable insights. We propose SeeUnsafe, a new framework that integrates Multimodal Large Language Model (MLLM) agents to transform video-based traffic accident analysis from a traditional extraction-then-explanation workflow to a more interactive, conversational approach. This shift significantly enhances processing throughput by automating complex tasks like video classification and visual grounding, while improving adaptability by enabling seamless adjustments to diverse traffic scenarios and user-defined queries. Our framework employs a severity-based aggregation strategy to handle videos of various lengths and a novel multimodal prompt to generate structured responses for review and evaluation and enable fine-grained visual grounding. We introduce IMS (Information Matching Score), a new MLLM-based metric for aligning structured responses with ground truth. We conduct extensive experiments on the Toyota Woven Traffic Safety dataset, demonstrating that SeeUnsafe effectively performs accident-aware video classification and visual grounding by leveraging off-the-shelf MLLMs. Source code will be available at \url{https://github.com/ai4ce/SeeUnsafe}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。