用PyTorch打造事件相机数据生成工具,助力神经形态视觉研究
A PyTorch-Enabled Tool for Synthetic Event Camera Data Generation and Algorithm Development
- 基于PyTorch构建可微分数字孪生,将图像转为事件数据
- 生成逼真事件数据,与真实设备输出匹配度高
- 适合神经形态视觉、事件相机算法研究者使用
事件相机(或称类脑相机)通过异步报告亮度显著变化来编码自然场景,相比传统相机具备更高动态范围、更优时间分辨率和更低数据带宽。然而其在特定领域研究中的应用受限于商业产品稀缺、缺乏现有数据集,以及非线性光学编码、独特噪声模型和张量化数据处理需求带来的建模挑战。为此,我们提出基于Python的可微分事件数据生成与处理库SENPI,支持从强度数据生成事件表示,并能有效处理前向模型中的非光滑与非线性问题。该库提供事件输入输出、处理、过滤与可视化模块,构建高效可扩展的工作流,适用于合成与真实事件数据。通过对比合成数据与真实事件相机输出,验证了其生成的真实性;并用于分析不同噪声条件下的相机行为,优化事件对比度阈值以提升编码效果。SENPI旨在降低研究门槛,成为神经形态视觉系统研究的重要工具。
原文摘要 · Abstract (English)
Event, or neuromorphic cameras, offer a novel encoding of natural scenes by asynchronously reporting significant changes in brightness, known as events, with improved dynamic range, temporal resolution and lower data bandwidth when compared to conventional cameras. However, their adoption in domain-specific research tasks is hindered in part by limited commercial availability, lack of existing datasets, and challenges related to predicting the impact of their nonlinear optical encoding, unique noise model and tensor-based data processing requirements. To address these challenges, we introduce Synthetic Events for Neural Processing and Integration (SENPI) in Python, a PyTorch-based library for simulating and processing event camera data. SENPI includes a differentiable digital twin that converts intensity-based data into event representations, allowing for evaluation of event camera performance while handling the non-smooth and nonlinear nature of the forward model The library also supports modules for event-based I/O, manipulation, filtering and visualization, creating efficient and scalable workflows for both synthetic and real event-based data. We demonstrate SENPI's ability to produce realistic event-based data by comparing synthetic outputs to real event camera data and use these results to draw conclusions on the properties and utility of event-based perception. Additionally, we showcase SENPI's use in exploring event camera behavior under varying noise conditions and optimizing event contrast threshold for improved encoding under target conditions. Ultimately, SENPI aims to lower the barrier to entry for researchers by providing an accessible tool for event data generation and algorithmic developmnent, making it a valuable resource for advancing research in neuromorphic vision systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。