iFinder让大模型看行车记录仪视频时更懂交通逻辑,零样本下事故判断准确率提升39%。
iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
- 将行车视频转为分层可解释数据结构,分离感知与推理过程。
- 利用物体朝向和全局上下文信息,零样本事故推理准确率最高提升39%。
- 无需训练,适合需要可解释性的后置驾驶视频分析场景。
将大语言模型(LLM)用于后置行车记录仪视频分析等领域任务时,受限于其通用训练背景及缺乏结构化先验知识,难以有效建模。由于此类分析通常仅依赖视觉信息(无激光雷达、GPS等),现有视觉-语言模型(V-VLMs)在空间推理、因果推断和事件可解释性方面表现不佳。为此,我们提出iFinder,一种结构化语义定位框架,通过将感知与推理解耦,将行车视频转化为适用于LLM的层次化可解释数据结构。iFinder作为模块化、免训练的流水线,采用预训练视觉模型提取关键线索——物体姿态、车道位置与物体轨迹,并将其分层组织为帧级与视频级结构。结合三阶段提示策略,支持LLM进行逐步、有依据的推理,以优化同行V-VLM的输出并提供精准判断。在四个公开行车视频基准上的评估表明,iFinder引入领域特定线索(尤其是物体朝向与全局上下文)的定位机制,在四项零样本驾驶基准上显著优于端到端V-VLMs,事故推理准确率最高提升39%。通过使用驾驶领域特异性表征对齐LLM,iFinder为后置驾驶视频理解提供了零样本、可解释且可靠的替代方案。
原文摘要 · Abstract (English)
Grounding large language models (LLMs) in domain-specific tasks like post-hoc dash-cam driving video analysis is challenging due to their general-purpose training and lack of structured inductive biases. As vision is often the sole modality available for such analysis (i.e., no LiDAR, GPS, etc.), existing video-based vision-language models (V-VLMs) struggle with spatial reasoning, causal inference, and explainability of events in the input video. To this end, we introduce iFinder, a structured semantic grounding framework that decouples perception from reasoning by translating dash-cam videos into a hierarchical, interpretable data structure for LLMs. iFinder operates as a modular, training-free pipeline that employs pretrained vision models to extract critical cues -- object pose, lane positions, and object trajectories -- which are hierarchically organized into frame- and video-level structures. Combined with a three-block prompting strategy, it enables step-wise, grounded reasoning for the LLM to refine a peer V-VLM's outputs and provide accurate reasoning. Evaluations on four public dash-cam video benchmarks show that iFinder's proposed grounding with domain-specific cues, especially object orientation and global context, significantly outperforms end-to-end V-VLMs on four zero-shot driving benchmarks, with up to 39% gains in accident reasoning accuracy. By grounding LLMs with driving domain-specific representations, iFinder offers a zero-shot, interpretable, and reliable alternative to end-to-end V-VLMs for post-hoc driving video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。