首个面向事件传感器的图像信号处理研究，构建了配对数据集并提出新方法。
RGB-Event ISP: The Dataset and Benchmark
- 构建首个事件与RAW图像同步采集的数据集，含3373张图像和多场景参数。
- 设计标准ISP流程生成参考RGB帧，用于评估事件引导的优化效果。
- 提出简单事件引导的ISP方法，揭示该领域关键挑战与未来方向。
事件引导成像因其革新即时成像系统的潜力而受到广泛关注。然而，现有方法主要在后处理阶段增强RGB图像，忽视了事件传感器对图像信号处理器（ISP）带来的挑战及事件信息对重构ISP流程的潜在优势。为此，我们首次开展事件引导的ISP研究。首先，提出一个全新的事件-RAW配对数据集，采用新型但尚未公开的传感器，实现像素级对齐的事件与RAW图像记录，包含3373张分辨率2248×3264的RAW图像，覆盖24个场景、3种曝光模式和3种镜头。其次，设计常规ISP流水线以生成高质量参考RGB帧，包括去马赛克、白平衡、去噪和色彩空间转换，并以ColorChecker作为参考。第三，将现有可学习ISP方法分为三类，并选取多个方法在新数据集上训练与评估。最后，因无先例可循，提出一种简单的事件引导ISP方法并在数据集上测试。我们进一步指出该领域关键技术挑战与未来方向。据我们所知，这是首个聚焦事件引导ISP的研究，旨在激发社区关注。代码与数据集见：https://github.com/yunfanLu/RGB-Event-ISP。
原文摘要 · Abstract (English)
Event-guided imaging has received significant attention due to its potential to revolutionize instant imaging systems. However, the prior methods primarily focus on enhancing RGB images in a post-processing manner, neglecting the challenges of image signal processor (ISP) dealing with event sensor and the benefits events provide for reforming the ISP process. To achieve this, we conduct the first research on event-guided ISP. First, we present a new event-RAW paired dataset, collected with a novel but still confidential sensor that records pixel-level aligned events and RAW images. This dataset includes 3373 RAW images with 2248 x 3264 resolution and their corresponding events, spanning 24 scenes with 3 exposure modes and 3 lenses. Second, we propose a conventional ISP pipeline to generate good RGB frames as reference. This conventional ISP pipleline performs basic ISP operations, e.g.demosaicing, white balancing, denoising and color space transforming, with a ColorChecker as reference. Third, we classify the existing learnable ISP methods into 3 classes, and select multiple methods to train and evaluate on our new dataset. Lastly, since there is no prior work for reference, we propose a simple event-guided ISP method and test it on our dataset. We further put forward key technical challenges and future directions in RGB-Event ISP. In summary, to the best of our knowledge, this is the very first research focusing on event-guided ISP, and we hope it will inspire the community. The code and dataset are available at: https://github.com/yunfanLu/RGB-Event-ISP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。