用本地化神经符号框架自动挖掘自动驾驶罕见场景数据。
Semantic-Drive: Democratizing Long-Tail Data Curation via Open-Vocabulary Grounding and Neuro-Symbolic VLM Consensus
- 分两阶段：先用开放词汇检测定位关键区域，再用推理模型分析场景。
- 召回率达0.966，风险评估误差降低40%，优于单模型和CLIP。
- 可在消费级显卡运行，保护隐私，适合车企自建数据标注系统。
自动驾驶系统的发展受限于长尾场景训练数据的稀缺。尽管车队积累了海量视频日志，但识别罕见高危事件（如异常横穿、施工绕行）仍依赖人工，成本高昂。现有方案或依赖粗粒度元数据搜索，精度不足；或使用云端视觉语言模型，存在隐私与成本问题。本文提出Semantic-Drive，一种本地优先的神经符号数据挖掘框架。该方法将感知分为两步：(1) 通过实时开放词汇检测器（YOLOE）实现符号锚定，聚焦注意力；(2) 通过推理型VLM执行取证式场景分析。为抑制幻觉，引入“系统2”推理时对齐策略，采用多模型“裁判-探员”共识机制。在nuScenes数据集上对比Waymo Open Dataset（WOD-E2E）分类体系，Semantic-Drive召回率达0.966（对比CLIP的0.475），风险评估误差降低40%。系统可在消费级硬件（NVIDIA RTX 3090）上运行，提供安全可控的云外替代方案。
原文摘要 · Abstract (English)
The development of robust Autonomous Vehicles (AVs) is bottlenecked by the scarcity of "Long-Tail" training data. While fleets collect petabytes of video logs, identifying rare safety-critical events (e.g., erratic jaywalking, construction diversions) remains a manual, cost-prohibitive process. Existing solutions rely on coarse metadata search, which lacks precision, or cloud-based VLMs, which are privacy-invasive and expensive. We introduce Semantic-Drive, a local-first, neuro-symbolic framework for semantic data mining. Our approach decouples perception into two stages: (1) Symbolic Grounding via a real-time open-vocabulary detector (YOLOE) to anchor attention, and (2) Cognitive Analysis via a Reasoning VLM that performs forensic scene analysis. To mitigate hallucination, we implement a "System 2" inference-time alignment strategy, utilizing a multi-model "Judge-Scout" consensus mechanism. Benchmarked on the nuScenes dataset against the Waymo Open Dataset (WOD-E2E) taxonomy, Semantic-Drive achieves a Recall of 0.966 (vs. 0.475 for CLIP) and reduces Risk Assessment Error by 40% ccompared to the best single scout models. The system runs entirely on consumer hardware (NVIDIA RTX 3090), offering a privacy-preserving alternative to the cloud.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。