arXiv:2607.12293cs.CV2026-07

用稀疏注意力融合多模态信息,提升行人过街意图预测精度与实时性。

Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction

论文配图:Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction
图 1 · 摘自论文原文
  • 设计稀疏跨模态注意力机制,精准选择关键跨模态交互
  • 在JAAD和PIE数据集上分别达0.85和0.92的准确率
  • 每帧推理仅17.23毫秒,适合自动驾驶实时部署

行人过街意图预测是自动驾驶中的安全关键任务。现有方法常依赖单模态输入或密集的多模态融合策略,难以有效捕捉视觉与运动信息的互补性,且引入冗余跨模态交互。本文提出ADAPT(Adaptive Domain-Aware Pedestrian Crossing Transformer),一个联合建模局部与全局视觉上下文及时间运动动态的多模态框架。ADAPT处理四类空间对齐的视觉模态(RGB图像、局部深度图、全局语义图、全局深度图)以及自车速度、行人边界框和骨骼姿态信息,通过五个专用模块:共享权重的Swin Transformer V2主干提取视觉特征,跨模态引导注意力模块实现层次化视觉融合,基于Mamba的运动特征编码模块高效建模时序动态,稀疏跨模态注意力模块有选择地保留最具信息量的跨模态交互,以及基于视觉变换器的时序特征融合模块完成序列级预测。在JAAD和PIE基准数据集上的大量实验表明,ADAPT持续优于现有最先进方法,同时保持低计算复杂度。在JAAD上,其在JAADbeh上达到0.73的AUC,JAADall上达0.85;在PIE上准确率达0.92,AUC为0.90。此外,ADAPT每样本仅需17.23毫秒推理时间,为智能交通与自动驾驶应用提供了预测精度与实时部署效率的良好平衡。

原文摘要 · Abstract (English)

Predicting pedestrian crossing intention is a safety-critical task for autonomous driving, yet existing approaches often rely on single-modal inputs or dense multimodal fusion strategies that inadequately capture complementary visual and kinematic information while introducing redundant inter-modal interactions. We propose ADAPT (Adaptive Domain-Aware Pedestrian Crossing Transformer), a multimodal framework that jointly models local and global visual context together with temporal motion dynamics for accurate pedestrian crossing intention prediction. ADAPT processes four spatially aligned visual modalities, including RGB images, local depth maps, global semantic maps, and global depth maps, together with ego-vehicle speed, pedestrian bounding boxes, and skeleton pose information through five specialized modules: a weight-shared Swin Transformer V2 backbone for visual feature extraction, a Cross-Modality Guided Attention module for hierarchical visual fusion, a Mamba-based Motion Feature Encoding module for efficient temporal modeling, a Sparse Cross-Modal Attention module that selectively preserves the most informative inter-modal interactions, and a Vision Transformer-based Temporal Feature Fusion module for sequence-level prediction. Extensive experiments on the JAAD and PIE benchmark datasets demonstrate that ADAPT consistently outperforms existing state-of-the-art methods while maintaining low computational complexity. On JAAD, the proposed method achieves an AUC of 0.73 on JAADbeh and 0.85 on JAADall, while on PIE it achieves an accuracy of 0.92 and an AUC of 0.90. Furthermore, ADAPT performs inference in only 17.23 ms per sample, offering an effective balance between predictive accuracy and real-time deployment efficiency for intelligent transportation and autonomous driving applications.

行人预测多模态融合稀疏注意力自动驾驶

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。