用大模型加速多摄像头交通视频分析,提升事故调查效率。
TrafficLens: Multi-Camera Traffic Video Analysis Using LLMs
- 分步处理多摄像头视频,用前序输出引导后续分析
- 实测视频转文本速度提升4倍,准确率不变
- 适合交通管理、智能城市等需要快速响应的场景
交通摄像头在智慧交通系统中至关重要,多摄像头部署可提升执法、交通管理和行人安全能力。然而,海量视频数据的高效管理与分析面临挑战。现有大型语言模型(LLMs)虽擅长文本任务,但将视频转化为文本需依赖视觉-语言模型(VLM),过程耗时且影响实时性。为此,我们提出TrafficLens,一种针对多摄像头交通路口的定制化算法。该方法利用摄像头间的重叠覆盖区域,采用迭代策略:对不同镜头依次应用具有不同令牌限制的VLM,以先前输出作为后续提示,实现快速生成详细文本描述。同时,通过对象级相似度检测器智能跳过冗余的VLM调用。在真实数据集上的实验表明,TrafficLens可将视频转文本时间减少最高达4倍,同时保持信息准确性。
原文摘要 · Abstract (English)
Traffic cameras are essential in urban areas, playing a crucial role in intelligent transportation systems. Multiple cameras at intersections enhance law enforcement capabilities, traffic management, and pedestrian safety. However, efficiently managing and analyzing multi-camera feeds poses challenges due to the vast amount of data. Analyzing such huge video data requires advanced analytical tools. While Large Language Models (LLMs) like ChatGPT, equipped with retrieval-augmented generation (RAG) systems, excel in text-based tasks, integrating them into traffic video analysis demands converting video data into text using a Vision-Language Model (VLM), which is time-consuming and delays the timely utilization of traffic videos for generating insights and investigating incidents. To address these challenges, we propose TrafficLens, a tailored algorithm for multi-camera traffic intersections. TrafficLens employs a sequential approach, utilizing overlapping coverage areas of cameras. It iteratively applies VLMs with varying token limits, using previous outputs as prompts for subsequent cameras, enabling rapid generation of detailed textual descriptions while reducing processing time. Additionally, TrafficLens intelligently bypasses redundant VLM invocations through an object-level similarity detector. Experimental results with real-world datasets demonstrate that TrafficLens reduces video-to-text conversion time by up to $4\times$ while maintaining information accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。