融合RGB与事件相机数据,用自注意力提升无人机避障预测精度。
RGB-Event Fusion with Self-Attention for Collision Prediction
- 双分支编码器分别处理视觉与事件数据,通过自注意力融合特征。
- 在50Hz下,融合模型平均精度提升1%,远距离(>0.5m)提升达10%。
- 事件相机表现优于RGB,且量化后仍具高效性,适合嵌入式部署。
确保自主机器人在动态现实环境中安全运行,需具备鲁棒且实时的障碍物避让能力。本文提出一种神经网络框架,利用RGB与事件视觉传感器预测无人机与动态物体的碰撞时间与位置。架构包含两个独立编码分支分别处理各模态数据,随后通过自注意力机制进行融合以提升预测精度。为便于对比,采用ABCD[8]数据集支持单模态与融合方法的详细评估。在50Hz预测吞吐率下,融合模型相比单模态方法平均精度提升1%,距离超过0.5m时提升达10%,但带来71%的内存增加和105%的浮点运算量(FLOPs)增长。值得注意的是,事件模型在位置误差上优于RGB模型4%,时间误差低26%,且计算成本相近,具备竞争力。此外,我们对事件模型进行了1至8比特量化,评估预测性能与计算效率间的权衡。研究揭示了在机器人应用中使用RGB与事件相机进行多模态感知的权衡关系。
原文摘要 · Abstract (English)
Ensuring robust and real-time obstacle avoidance is critical for the safe operation of autonomous robots in dynamic, real-world environments. This paper proposes a neural network framework for predicting the time and collision position of an unmanned aerial vehicle with a dynamic object, using RGB and event-based vision sensors. The proposed architecture consists of two separate encoder branches, one for each modality, followed by fusion by self-attention to improve prediction accuracy. To facilitate benchmarking, we leverage the ABCD [8] dataset collected that enables detailed comparisons of single-modality and fusion-based approaches. At the same prediction throughput of 50Hz, the experimental results show that the fusion-based model offers an improvement in prediction accuracy over single-modality approaches of 1% on average and 10% for distances beyond 0.5m, but comes at the cost of +71% in memory and + 105% in FLOPs. Notably, the event-based model outperforms the RGB model by 4% for position and 26% for time error at a similar computational cost, making it a competitive alternative. Additionally, we evaluate quantized versions of the event-based models, applying 1- to 8-bit quantization to assess the trade-offs between predictive performance and computational efficiency. These findings highlight the trade-offs of multi-modal perception using RGB and event-based cameras in robotic applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。