融合激光雷达与事件相机,实现突发行人横穿的毫秒级避让。
DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance

- 用Transformer注意力融合激光雷达点云与事件相机流,动态感知高速变化。
- 在昼夜复杂光照下轨迹误差最低,控制命令更精准可靠。
- 适合高安全要求的自动驾驶系统研发,尤其关注突发障碍物应对。
当前端到端自动驾驶系统主要依赖帧式传感器,在高速动态场景(如突发行人横穿)中存在固有感知延迟和运动模糊问题。为解决这一关键安全隐患,本文提出DeepIPCv3,一种新型多模态自主导航框架,将激光雷达点云的密集三维空间结构与动态视觉传感器(DVS)的微秒级异步事件流相结合。设计基于Transformer的跨模态注意力机制,动态关联不同模态信息,使网络能即时优先处理高速动态更新,同时保持对场景结构的完整感知。融合后的潜在表示通过混合策略网络映射为安全局部路径点与可执行控制指令,结合启发式轨迹跟踪与直接神经预测。由于真实测试突发横穿场景存在严重物理风险,该框架在自建多模态数据集上进行离线评估,覆盖光照良好正午与挑战性黄昏条件。大量对比与消融实验表明,DeepIPCv3达到当前最优预测性能,有效消除误检与运动模糊,显著降低轨迹与控制命令误差,可在任何光照条件下实现高度响应、数学上可保证的规避动作。代码将开源至GitHub。
原文摘要 · Abstract (English)
Current end-to-end autonomous driving systems predominantly rely on frame-based sensors, which suffer from inherent perception latency and motion blur during highly dynamic encounters, specifically sudden pedestrian crossings. To address this critical safety vulnerability, we propose DeepIPCv3, a novel multi-modal autonomous navigation framework that synergizes the dense 3D spatial geometry of LiDAR point clouds with the microsecond-level asynchronous event streams of a Dynamic Vision Sensor (DVS). We introduce a Transformer-inspired cross-modal attention mechanism to dynamically correlate these distinct modalities, allowing the network to instantaneously prioritize high-speed dynamic updates without sacrificing structural scene awareness. The fused latent representations are then mapped to safe local waypoints and executable control commands via a hybrid policy network that blends heuristic trajectory tracking with direct neural predictions. Due to the severe physical risks associated with live testing of these sudden crossing scenarios, the framework is rigorously evaluated offline using a custom multi-modal dataset collected across both well-illuminated noon and challenging evening conditions. Extensive comparative and ablation studies demonstrate that DeepIPCv3 achieves state-of-the-art predictive performance. By effectively eliminating exposure failures and motion blur, the proposed LiDAR and DVS fusion yields the lowest trajectory and control command errors, enabling highly reactive, mathematically bounded evasive maneuvers regardless of ambient illumination. To support future research, we will release the codes to our GitHub repo at https://github.com/oskarnatan/DeepIPCv3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。